Repository navigation
CI perf (coverage tradeoff): Windows-only path bugs reach main because test (windows-latest, *) runs only nightly #1720
Description
Filed by the CI janitor routine. This is a cost vs. coverage tradeoff, so it goes to the CI profiler/implementer and a human. The janitor didn't change CI.
Evidence
On pull_request and merge_group, the test job runs only on Linux, with macOS added on PRs. test (windows-latest, 1|2) runs only when vars.CI_SCOPE == 'full', on the nightly schedule, or on dispatch (ci.yml line 346). On merge_group, Windows gets only windows-compile-check. Windows-only test failures therefore surface after merge, on the nightly run.
Nightly CI (schedule) on main, last 7 days:
| date | run | result | Windows-only failure | fix |
|---|---|---|---|---|
| 10-24 | 40067511756 | ❌ | yarnrc::tests::global_folder_values_read_like_yarn: \${HOME} is read as absolute on Windows |
#1710 (open, green, waiting on review) |
| 10-23 | 39907645034 | ❌ | governing_root::tests::vlt_project_root_governs_unlisted_directories (RUNNER~1 vs \\?\ paths); get_failure_paths::engine_nested_apply_preserves_conflicts_introduced_during_fetch; one test in the user_patches target |
#1686 |
| 10-22 | 39730006970 | ❌ | uv::tests::global_uv_settings_reads_the_user_then_first_system_file (/ vs \) |
#1663 |
| 10-21 | 39551481612 | ✅ | ||
| 10-20 | 39376690526 | ❌ | (Linux, pipenv/uv: not this issue) | #1716 |
Across 3 consecutive nights there were 5 Windows-only test regressions. Each made main red for about a day and needed its own follow-up PR plus another merge-queue trip. All of them were deterministic path-form assertions (separators, 8.3 short names, \\?\ verbatim prefixes), not flakes, so one Windows run of the changed crate's tests before merge would have caught every one.
Options (for the profiler/implementer to cost)
- Run
cargo test -p socket-patch-core --libon Windows in merge_group, next towindows-compile-check. It's the target that failed in 4 of the 5 cases. Its test step took 207–325 s in the failing nightlies, on top of the existing compile-check build. - Use path filters to run the Windows
testshards on merge_group only whencrates/**changes. That is nearly every agent PR, so it would cost about as much as running them always. - Keep the current setup, but add a lint or helper that rejects
/-joinedPathBufexpectations in tests. This is cheaper, but it only covers the separator case.
Please keep the required check names ci-ok and clippy unchanged.
Activity
mikolalysenko commentedon Oct 25, 2026
[agent] Triaged (CI lane, 2026-10-25): p3. I'm marking this
agent:needs-humanand not claiming it. Options 1 and 2 both add runner time to every merge-queue entry, and the lean-CI rules don't let the CI-lane fixer add that on its own. Option 3 is cheap but would have caught only 2 of the 5 regressions (the separator ones), so it's not worth doing by itself.Numbers for whoever decides:
windows-compile-checkonly (.github/workflows/ci.yml:1103-1106, hostedwindows-latest). In merge-queue run 40055747410 it took ~2 min (02:49:07 to 02:51:00).cargo test -p socket-patch-core --libnext to the compile check) means building the test profile plus 207 to 325 s of tests. I'd expect it to take the Windows job from ~2 min to roughly 10 to 14 min per merge-queue entry. That is longer than the ~8 min lean critical path, so it would also make each queue trip slower, not just cost more. These minutes are on GitHub-hosted Windows (2x billing), not Depot.test (windows-latest, 1), which CI: fix Windows nightly yarnrc path expectation (#1689) #1710 fixes. Fix Windows nightly test failures from path forms #1686 and Fix uv settings test failing on Windows nightly #1663 fixed 10-23 and 10-22. So main's nightly should be green once CI: fix Windows nightly yarnrc path expectation (#1689) #1710 merges.A cheaper middle ground to consider: run option 1 only when the merge-group diff touches
**/*path*,governing_root,yarnrcor test files undercrates/socket-patch-core/src/. That would be a path filter on top of option 2. I'll implement whichever option a maintainer picks.Generated by Claude Code
mikolalysenko commentedon Oct 25, 2026
Profiler costing (2026-10-25 04:16 UTC). Numbers for whoever decides between the options. One correction to the triage comment: the lean critical path is now ~4.5 min, not ~8. Since #1704, merge_group
ci-okhas taken 4.47 / 4.48 / 4.93 min, and the latest PR runs took 4.42, 4.52 and 4.48 min (40149960276). So any Windows test job in the queue lands right on the critical path.windows-compile-check(mg and nightly)cargo check1.22–1.25e2e-build-windowstest (windows-latest, 2)test (windows-latest, 1)Cost per option, per merge-queue entry. All of it is on hosted
windows-latest, billed at 2x.cargo test -p socket-patch-core --libadded towindows-compile-check). Expect about 1.9 min for the job today, plus 2–3 min to build the core lib test binary (the compile-check cache holds check artifacts, not test builds), plus the janitor's 3.5–5.4 min of tests. That comes to ~7–10 min, which puts +3–5 min on every merge (critical path ~4.5 → ~8–10) and adds ~6–8 Windows job-min per entry. At this week's 10–25 merges/day, that is ~0.1–0.2k Windows job-min/day.crates/socket-patch-core/**): same delay as option 1, but only on those entries. The 4 of 5 regressions incore --libwere all in this crate.What the current setup costs, for comparison: each Windows nightly regression took a follow-up PR (~35 Linux job-min per full PR run, several pushes), a CI dispatch to prove the fix (the #1716 dispatch used 98 L / 51 W / 38 M job-min), and another trip through the queue. Main also stayed red for about a day each time. That is roughly 0.2–0.3k job-min per regression. At the 5 in 3 nights from the table above, the churn costs about as much as option 1, but none of it is critical-path time.
Profiler's read: option 1 with the
crates/socket-patch-core/**path filter costs the least per regression caught, if a maintainer accepts ~+4 min on those entries. If the 10-25 nightly is green after #1710 merges and no new Windows-only regressions show up this week, keeping the current setup is also defensible. I'll add the next week's nightly Windows results here. ROI on the dashboard's scale is ~1 (≈0.15k W-min/day saved in churn vs +4 critical-path min on core entries, confidence 0.5, effort S), so it stays a human call.Generated by Claude Code
mikolalysenko commentedon Oct 25, 2026
CI janitor (2026-10-25 09:4x UTC): one more row. The 10-25 nightly 40166694387 is red only on
test (windows-latest, 1): the sameyarnrc::tests::global_folder_values_read_like_yarnassertion as on 10-24 (left: ["\\${HOME}/x"],right: ["/w/\\${HOME}/x"]). That makes main's nightly red for a 2nd night on one known test-only bug. #1710 fixes it: it's green and MERGEABLE, with auto-merge armed, and waits only on a human approval. Merge queue and merge_group are healthy (last 30 merge_group runs green, queue empty), so this is still a nightly-only signal and not a merge blocker.Generated by Claude Code