Repository navigation
CI performance dashboard #1182
Description
Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.
Headline (run 2026-10-14 04:16 UTC)
Window: 2026-10-13 04:16 → 2026-10-14 04:16. 44 PRs merged. CI ran 56 merge_group runs: 45 success, 1 failure and 10 cancelled rebuilds, all from the 13:45 eviction. There was no merge_group run between 04:16 and 13:11. No new issues this run. The Depot slowdown in #1512 looks over since about 01:00. The slow-start share fell from 57% to 15% on 16-core hosts and from 24% to 13% on 4-core hosts. The 4 merge_group runs since 01:48 reached ci-ok in 7.1–8.6 min, and their PRs went from enqueue to merge in 8.2–8.9 min. Fresh numbers are on the issue. It stays open until a weekday-load sample confirms the recovery.
Sample: 583 runs in 24h. The run list is capped at 3 pages (14:15 onward), plus an event-filtered list of all 224 merge_group runs across the 4 merge_group workflows. Job detail (filter=all) was fetched for 118 runs:
- all 56 CI merge_group runs
- 46 CI pull_request runs (every other one)
- 16 push runs
That is 4,928 jobs. Merge-queue timelines were read for 15 PRs. No job logs were read, because no new candidate needed in-step detail. GraphQL is still blocked (HTTP 403). The run used about 160 REST requests.
| metric | 24h |
|---|---|
| PRs merged | 44 |
| Merge queue enqueue → merge | p50 10.1 / p90 11.5 min (n=15, range 8.2–12.7). • 22:49 and 23:49 batches: 9.8–11.5 • Since 01:48: 8.2–8.9 Batches of 2–4 PRs, so max_entries_to_build 5 didn't bind. |
| Evictions | 0 CI evictions since the 13:45 flake. #1505 left the queue at 22:59 and was re-enqueued at 23:49 (no CI failure of its own). |
merge_group created → ci-ok |
p50 8.57 / p90 11.02 (n=45). • 15:20–00:00 (slow hosts): 8.6–12.2 • Since 01:48: 7.12, 7.77, 7.98, 8.55 Last job before ci-ok: coverage in 26 runs, the Gradle e2e leg in 19. |
PR run created → ci-ok |
p50 9.85 / p90 12.12 (n=18 full runs, 45.3 job-min each). 28 of 46 sampled runs were stubs with no billable job-minutes. |
push to main created → ci-ok |
p50 4.45 / p90 5.93 (n=16, 11.0 job-min per run) |
| Nightly | 38353589047 (10-13 05:51) was green: 244 L / 82 W / 64 M job-min (carried over). The 10-14 nightly has not run yet. |
| Linux queue wait p50 / p90 / max | Depot 0.02 / 0.02 / 0.55 (n=2,300) |
| macOS queue wait | No macOS job ran on PR, merge_group or push. Only the nightly runs macOS. |
| Windows queue wait p50 / p90 / max | 0.03 / 0.07 / 0.58 (windows-compile-check, n=56) |
| Job-min/day (Linux / Windows / macOS) | ~5.5k / 0.22k / 0.064k. By event: • merge_group: 2.13k L + 0.14k W (38.0 L per run; 33.0–33.6 at 03:49) • CI pull_request: ~2.9k (17.9 per run including stubs, ~160 runs/day) • push: ~0.3k • nightly: 244 / 82 / 64 |
Depot slow-start share by period (a job counts as slow when its first step starts more than 15 s after the job, #1512):
| period (UTC) | 16-core | 4-core | Run tests with coverage p50 |
clippy p50 |
|---|---|---|---|---|
| 10-13 13:00–15:20 | 20% (22/111) | 23% | 330 s | 1.40 |
| 15:20–22:00 | 59% (53/90) | 70% | 352 s (407–480 s on slow hosts) | 2.17 |
| 22:00–01:00 | 57% (24/42) | 24% | 402 s | 1.47 |
| 10-14 01:00–04:16 | 15% (4/27) | 13% | 315 s | 1.37 |
Top jobs by sampled minutes:
coverage: 517 min, p50 5.95 / p90 8.25- Gradle
e2eleg: 358 min, p50 4.73 / p90 7.02 e2e_vex_build pip: 201 min, p50 2.70e2e_redirect_npm_build: 200 min, p50 2.63hosted-e2e: 177 min, p50 1.60clippy: 158 min, p50 1.47 / p90 2.33windows-compile-check: 144 min, p50 2.63e2e-build: 136 min, p50 1.27
Top steps: Run e2e tests 889 min, Run tests with coverage 434, Checkout 149, Set up job 98, Type-check the CLI and every CLI test target 97, Run hosted-mode production e2e 96, Install Rust 92.
Critical path (unchanged): clippy → coverage → ci-ok, in parallel with e2e-build → Gradle e2e leg → ci-ok. On healthy hosts the two paths finish within about 1 min of each other, so shortening one alone saves less than 1.5 min.
Other items watched, not filed (below the bar or not enough evidence):
Install Rust: p50 7 s and p90 10 s across 726 jobs, about 130 Linux job-min/day. A pre-baked toolchain would save about 5 s per job, off the critical path. That doesn't pay for M effort.Cache cargo: p50 3 s and p90 9 s. Not a problem.- Shard
coverage/ split the Gradle leg: ~1–1.5 min per merge, and both paths would need it (see above). Below the bar. clippy→coverageserialization: ~1.4 min at healthy host speed. Below the bar, and M effort.- Release dispatches:
Releaseran 5 times, andPublish crates.io 5.0.1ran once. That is release work, not CI time. - CI: skip PR re-test after a clean main merge (#1373) #1383 (
pr_unchanged): not re-checked this run.
Merge-queue settings (ruleset 24668460, read 04:2x): unchanged.
max_entries_to_build5 andmax_entries_to_merge5min_entries_to_merge_wait_minutes5- ALLGREEN, timeout 90, SQUASH
- Required checks:
ci-okandclippy
History
| run (UTC) | merges/24h | MQ enq→merge p50/p90 (min) | mg CI success p50 | PR push→ci-ok p50/p90 | mac queue p50/p90 | linux queue p50/p90 | job-min/day L/W/M |
|---|---|---|---|---|---|---|---|
| 2026-10-14 04:16 | 44 PRs merged (45 mg CI successes 13:11–03:49, 1 failure, 10 cancelled) | 10.1 / 11.5 (n=15; 8.2–8.9 since 01:48) | 8.6 (n=45, p90 11.0; 7.1–8.6 since 01:48, n=4) | 9.85 / 12.12 (n=18 full runs, created→ci-ok) | none ran | 0.02 / 0.02 (Depot, n=2,300) | ~5.5k / 0.22k / 0.064k (mg 38.0 L / 2.6 W per run; 33 at 03:49; Depot slow-start share fell to 13–15% after 01:00) |
| 2026-10-14 00:16 | ~41 (41 mg CI successes 13:11–23:49; 40 PRs merged since 00:16; 0 mg runs 00:16–13:11) | 11.3 / 12.0 (n=15; 9.8–12.7) | 8.7 (n=41, p90 11.0; 9.5 since 20:16, n=8) | 9.92 / 12.08 (n=19 full runs, created→ci-ok) | none ran | 0.02 / 0.02 (Depot, n=2,188) | ~5.8k / 0.21k / 0.064k (mg 36.7 L / 2.6 W per run; 16-core hosts still slow, 4-core recovered at 22h) |
| 2026-10-13 20:16 | ~39 (39 mg CI successes, 1 failure, 10 cancelled) | 11.4 / 15.6 (n=15; 9.7–12.7 outside the 13:5x eviction) | 8.6 (n=39; 10.8 since 15:49, n=8) | 10.63 / 12.07 (n=13 full runs since 16:20, created→ci-ok) | none ran | 0.02 / 0.02 (Depot, n=1,980) | ~4.7k / 0.21k / 0.064k (mg per run 36 → 52 L job-min after 15:20: slow Depot hosts) |
| 2026-10-13 16:17 | ~39 (39 mg CI successes; 14-entry burst 13:11–13:35; 1 flake eviction 13:45 → 10 cancelled rebuilds) | 11.6 / 23.7 (n=18; burst entries 6–14 10.4–18.1, eviction victims 23.7–25.3) | 8.38 (n=27, p90 10.83) | 8.02 / 11.67 (n=14 full runs, created→ci-ok) | none ran (no macOS on PR/mg/push) | 0.02 / 0.02 (Depot, n=1,620) | ~4.8k / 0.21k / 0.064k (mg 36.4 / 2.6 / 0 per run + 340 L job-min in cancelled rebuilds) |
| 2026-10-13 12:16 | ~33 (37 mg runs, carried; 0 merges since 22:57) | 8.8 / 10.5 (n=15, carried) | 8.22 (carried) | 7.57 / 7.87 (n=5 fresh full runs, created→ci-ok) | 0.13 / 0.17 (nightly only, carried) | 0.02 / 0.02 (Depot, n=145) | ~3.9k / 0.18k / 0.064k (Release dispatches mostly aged out; mg 35.4 / 2.6 / 0 per run) |
| 2026-10-13 08:16 | ~33 (37 mg runs, carried; 0 merges since 22:57) | 8.8 / 10.5 (n=15, carried) | 8.22 (n=33, carried) | 7.6 / 9.0 (n=6 fresh full runs, created→ci-ok) | 0.13 / 0.17 (nightly only) | 0.02 / 0.02 (Depot) | ~4.1k / 0.25k / 0.10k (incl. Release dispatches; ~3.95k / 0.18k / 0.064k without; nightly green, 244 / 82 / 64) |
| 2026-10-13 04:16 | ~33 (37 mg runs; 0 merges since 22:57; 1 eviction, carried) | 8.8 / 10.5 (n=15, carried) | 8.22 (n=33, p90 9.21) | 8.02 / 10.04 (n=10 full runs, created→ci-ok) | 0.12 / 0.15 (nightly + Release only) | 0.02 / 0.02 (Depot, n=1,828) | ~4.1k / 0.22k / 0.07k (incl. 3 Release dispatches; ~3.95k / 0.15k / 0.035k without; mg 35.4 / 2.6 / 0 per run; push ~8.7 per run after #1415) |
| 2026-10-13 00:16 | 35 (40 mg runs; 1 eviction, carried; release 5.0.0) | 8.8 / 10.5 (n=15, max 10.5, 0 evictions) | 8.15 (n=36, p90 9.2) | 8.43 / 9.95 (n=10 full runs, created→ci-ok) | 0.12 / 0.15 (nightly + dispatch only) | 0.02 / 0.02 (Depot, n=2,309) | ~5.4k / 0.35k / 0.16k (incl. 3 CI + 3 Release dispatches; ~4.9k / 0.16k / 0.03k without; mg 35.1 / 2.7 / 0 per run; push ~8.7 per run) |
| 2026-10-12 20:16 | 36 (42 mg runs; 15:44 burst drained by 16:19; 1 eviction) | 9.7 / 28.8 (n=15; burst entries 27.6–31.9, others 7.9–10.4) | 7.93 (n=38, p90 9.03; 8.43 since 16:16) | 7.67 / 10.22 (n=8 full runs, created→ci-ok) | 0.12 / 0.17 (nightly + dispatch only) | 0.02 / 0.02 (Depot, n=2,534) | ~4.4k / 0.32k / 0.14k (incl. 3 CI + 2 Release dispatches; ~4.05k / 0.17k / 0.03k without; mg 33.9 / 2.9 / 0 per run; push now ~8 per run) |
| 2026-10-12 16:16 | 31 (42 mg runs; 11-entry burst at 15:44; 1 eviction) | 8.9 / 25.7 (n=15; burst entries 6–10 25.2–26.4) | 7.63 (n=34, p90 8.67) | 7.93 / 9.05 (n=9 full runs, created→ci-ok) | 0.12 / 0.17 (nightly + dispatch only) | 0.02 / 0.02 (Depot, n=2,854) | ~4.4k / 0.32k / 0.12k (incl. 3 dispatches; ~4.1k / 0.20k / 0.03k without; mg 32.3 / 3.6 / 0 per run) |
| 2026-10-12 12:26 | 23 (23 mg runs; no merges since 02:57; timelines carried) | 8.0 / 8.3 (n=15, carried) | 7.33 (n=23, p90 9.2) | 7.38 / 8.9 (n=12 full runs, created→ci-ok; 1 flaky coverage re-run) | 0.13 / 0.17 (nightly only) | 0.02 / 0.02 (Depot, n=2,035) | ~3.9k / 0.34k / 0.19k (incl. 3 dispatches; ~3.2k / 0.18k / 0.06k without; mg 30.7 / 4.6 / 0 per run) |
| 2026-10-12 08:16 | 23 (23 mg runs; 15 timelines) | 8.0 / 8.3 (n=15, max 8.7, 0 evictions) | 7.3 (n=19 after #1386, p90 7.9) | 7.57 / 7.9 (n=9 full runs, created→ci-ok; 1 coverage re-run to 15.9) | 0.12 / 0.17 (nightly + dispatch only) | 0.02 / 0.02 (Depot, n=2,659) | ~3.9k / 0.34k / 0.19k (incl. 3 dispatches; ~3.2k / 0.18k / 0.06k without; mg 30.7 / 4.6 / 0 per run) |
| 2026-10-12 04:17 | 23 (23 mg runs; 15 timelines) | 8.0 / 8.3 (n=15, max 8.7, 0 evictions) | 7.3 (n=19 after #1386, p90 7.9) | 7.45 / 8.9 (n=12 full runs, created→ci-ok) | 0.12 / 0.15 (nightly + dispatch only) | 0.02 / 0.02 (Depot, n=2,645) | ~4.0k / 0.33k / 0.19k (incl. 3 full-matrix dispatches; ~3.3k / 0.17k / 0.05k without; mg 30.8 / 4.4 / 0 per run) |
| 2026-10-12 00:16 | 21 (20 mg runs, 1 in flight; 15 timelines) | 7.9 / 8.3 (n=15, max 8.7, 0 evictions) | 7.3 (n=16 after #1386, p90 7.9) | 7.37 / 7.4 (n=18 full runs, created→ci-ok) | 0.12 / 0.17 (nightly only) | 0.02 / 0.02 (Depot, n=2,075) | ~2.7k / 0.16k / 0.05k (mg 30.2 / 4.7 / 0 per run) |
| 2026-10-11 16:17 | 15 (weekend; 15 mg runs, 1 merge since 01:02) | 10.3 / 18.9 (n=15, max 19.1: 17:31 burst of 8, entries 6–8 waited a 2nd build round) | 9.65 (n=15, p90 9.81) | 7.35 / 7.8 (n=10 full runs, created→ci-ok) | 0.12 / 0.15 (nightly only) | 0.02 / 0.02 (Depot) | ~1.9k / 0.18k / 0.05k (mg 28.5 / 7.2 / 0 per run) |
| 2026-10-11 20:16 | 16 (15 timelines; 9 merges after #1386) | 8.7 / 10.6 (n=15); 7.9 / 8.7 after #1386 (n=9) | 7.3 after #1386 (n=9, p90 7.98); 9.65 before (n=7) | 7.37 / 7.4 (n=10 full runs, created→ci-ok) | 0.12 / 0.15 (nightly only) | 0.02 / 0.02 (Depot, n=1,668) | ~2.1k / 0.16k / 0.05k (mg 30.6 / 5.9 / 0 per run) |
| 2026-10-11 12:17 | 28 (weekend; 30 mg runs) | 10.3 / 10.7 (n=14, max 18.8 burst) | 9.6 (n=30, p90 9.95) | 7.2 / 8.1 (n=25, created→ci-ok) | 0.12 / 0.18 (nightly/dispatch only) | 0.02 / 0.02 (Depot) | ~6.0k / 0.8k / 0.5k (mg 29 / 7.5 / 0 per run) |
| 2026-10-09 20:16 | ≥99 (8 since 16:16) | last 78 / 112 (n=6, Linux backlog) | 79.6 (n=6, 77–81) | 23.3 / 65.5 (n=12; 51–79 after 17:45) | 0.87 / 4.08 | 5.9 / 18.3 mg (16.6 p50 at 19h) | ~150k / ~35k / 13.6k (carried over; CI per-run unchanged: mg 312 / 69 / 60) |
Open ci-perf issues by ROI
ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).
| ROI | issue | saving | status |
|---|---|---|---|
| ~1.75 | #1512 (settings) slow Depot 16-core hosts | ~1.5 min p50 off the merge-queue critical path and PR feedback, plus ~0.5k Linux job-min/day | Looks recovered since 01:00 (16-core slow share 15%, mg → ci-ok 7.1–8.6). Close after 1–2 more confirming runs. |
| ~1.0 | #1248 (settings) merge-queue build concurrency | ~2–10 min for each entry past the 5th in a burst | Needs a human decision. No burst above 5 entries in this window (largest batch 4), so no comment. |
Open ci-perf / ci-janitor PRs: none.
Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect
| PR | merged | before → after |
|---|---|---|
| #1497 fix phantom version bump in release-readiness (ci-janitor) | 10-13 19:59 | Correctness only. No effect on timing. |
#1484 hermetic spawns in vendor build tests; serializes the SOCKET_NPM_REGISTRY test |
10-13 14:59 | The flake from #1466 failed coverage 2× at 13:45–14:23 (1 eviction, 10 cancelled rebuilds, 340 job-min) → 0 coverage failures in the 33 merge_group runs since 14:16 (through 10-14 03:49). Delivered. |
| #1459 ride out crates.io blips in required cargo e2e legs (ci-janitor) | 10-13 13:21 | Reliability only. 0 cargo e2e failures since. |
| #1415 run the full e2e tier nightly, not per push | 10-12 16:11 | Push run: 50 jobs and 38.7–42.7 job-min (n=3, 15:53–16:10) → 9 jobs and 7.2–11.3 job-min (n=14, mean ~8.7, through 22:57). Push created → ci-ok 4.3–5.25 → 3.1–4.85 min. That is ~−0.45k Linux job-min/day at 21 pushes/day, better than the ~−300 expected. Delivered. |
| #1438 annotate failing tests in the coverage job (ci-janitor) | 10-12 17:58 | Diagnostics only. |
| #1418 env race flake in cache_env scrub self-tests (ci-janitor) | 10-12 16:10 | 0 coverage failures in merge_group after 16:02 (n=23 through 22:49). #1416 is closed. |
| #1411 macOS go.work member match (ci-janitor) | 10-12 16:18 | The 10-13 nightly 38353589047 was green, including test (macos-latest, 1), which failed on 10-12. Delivered. |
| #1392 Windows flake in notifier dead-endpoint test (ci-janitor) | 10-11 19:23 | 0 Windows failures in merge_group since. |
| #1386 type-check Windows in the queue, in parallel | 10-11 18:21 | mg created → ci-ok 9.65 → 7.6–7.9 min. e2e-build-windows hasn't been last before ci-ok in this window. Delivered. |
| #1383 skip PR re-test after a clean main merge | 10-11 18:01 | Not yet observable. 0 of 231 sampled PR runs across five profiles took the path. |
| #1387 re-download truncated e2e-bin archives (ci-janitor) | 10-11 17:50 | 0 e2e-bin download failures. |
| #1376 dead Depot runner doesn't cancel a merge-queue run | 10-10 13:37 | 0 runner losses. |
| #1355 reuse the merge-queue verdict on main, trim coverage/sbt | 10-10 08:57 | Push-to-main run 22+ → 3.4 min. Delivered. |
| #1375 lean per-package-manager gate | 10-09 21:29 | Job-min per mg run 312 / 69 / 60 → 33.9 / 2.9 / 0. PR → ci-ok 21.8 → 7.7. Delivered. |
| #1362 Depot Linux runners | 10-09 20:53 | Linux queue wait p90 18.3 → 0.02 min. Delivered. |
| #1284, #1275 compat path filters | 10-09 | No compatibility workflow ran in the window. Delivered. |
Method notes
- GraphQL is blocked in this environment, so merge-queue state comes from REST timelines of merged PRs. The timelines use the last enqueue before the merge.
- Jobs are fetched with
filter=all. Queue wait = jobstarted_at−created_at; skipped jobs are excluded. "Start-up gap" = jobstarted_at→ first stepstarted_at; over 15 s counts as a slow host. - Critical path = merge_group run
created_at→ci-okcompleted_at. - Daily job-min = per-run mean × 24h run count per event. The PR and push counts are scaled from the 3 listed pages (14:15–04:04): 92 PR runs and 16 push runs over ~13.8 h.
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 8, 2026
[agent] There are now two open dashboard issues: this one and #1175, both titled "CI performance dashboard" and labeled
ci-perf-dashboard. The body says it is maintained in place, so the profiler routine should update one issue rather than open a new one each run. I'm not closing either one, because I can't tell which issue the routine will update next. Whoever owns the profiler should close the copy it no longer updates.Generated by Claude Code