Hook fleet (5) chaos isolation fixtures by EdbertChan · Pull Request #1336 · EdbertChan/catstack

EdbertChan · 2026-10-11T16:25:55Z

Summary

When one hook check breaks, the others still report, and the broken one is logged as a crash.

Before, a check that threw an error was logged as a handled error. That hid real crashes from the hook health numbers.

Now a thrown error is logged with a failing exit code. A new test runs five checks at once: one throws, one hangs, one quits early, and two behave.

Review Claim

Fixture detectors that raise, hang past the soft budget, and exit non-zero each produce their own failure row, while every sibling detector's verdict and row is still emitted.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Hook stdout and exit codes stay the same. The dispatcher still exits 0 and still prints sibling verdicts; only the metrics row for a raising detector changes, from caught_error with exit code 0 to crashed with exit code 1.

Slice Rationale

This is claim 5 of the hook fleet dispatcher work. It needs the in-process dispatcher from #1335, so this branch sits on top of that one and its diff against main also shows that slice until #1335 lands.

This slice's own commit is 6d192023: one line in the dispatcher, four fixture detectors, and one test. Keeping it apart lets a reviewer check failure isolation without re-reading the dispatcher.

Non-goals

No change to the dispatcher design, the event parse, or the shared transcript parse from Hook fleet (4) SubagentStop dispatcher #1335.
No change to the event uid, replay harness, or scorecard dashboard slices.
No change to how hang or non-zero exit rows are classified; the test only pins them.
The fixture detectors are never installed. They live under the runner's test folder and are registered only in a scratch manifest inside the test.

Measured

Command: cd engine/hooks/_runner && /opt/homebrew/bin/python3.12 -m unittest tests.test_dispatch -k chaos

base: the same test run against the dispatcher from the parent commit 67ed8460

AssertionError: 'caught_error' != 'crashed'
- caught_error
+ crashed

Ran 1 test in 1.181s

FAILED (failures=1)

head: commit 0b871478

test_sdk_chaos_failures_each_emit_rows_without_silencing_siblings (tests.test_dispatch.DispatchCLI.test_sdk_chaos_failures_each_emit_rows_without_silencing_siblings) ... ok

Test Plan

cd engine/hooks/_runner && /opt/homebrew/bin/python3.12 -m unittest tests.test_dispatch -v — Ran 21 tests in 10.165s, OK
Same chaos test with the parent commit's dispatcher swapped in, in a scratch copy — fails with 'caught_error' != 'crashed', as shown under Measured

Run on Python 3.12.14, the version CI uses. The macOS system Python 3.9.6 cannot run the two in-process detector tests: they fail with ModuleNotFoundError: No module named 'tomllib'.

Revert Plan

Safe to revert? Yes
Revert command: git revert 6d192023
Post-revert steps: None. A raising detector goes back to being logged as caught_error.
Data migration? No. Rows already written keep their recorded outcome.

🤖 Generated with Claude Code

Note

Low Risk
One-line metrics classification change for in-process SDK errors; harness stdout/stderr and multi-hook merge behavior are explicitly pinned by tests and described as unchanged.

Overview
In-process SDK hook failures are now recorded as real crashes in metrics instead of caught_error with exit code 0. In _run_sdk_one, non-SystemExit exceptions caught from detect() now set exit_code to 1 (unchanged behavior for SystemExit codes and timeouts).

Adds chaos test fixtures (raise, hang, SystemExit(3), healthy ok) and test_sdk_chaos_failures_each_emit_rows_without_silencing_siblings, which runs five SubagentStop SDK hooks together and asserts each failure gets its own row (crashed / timed_out, dispatch_path sdk_error) while healthy siblings still spoke and the overall dispatch still exits 0.

^{Reviewed by Cursor Bugbot for commit 92fb3ae. Bugbot is set up for automated code reviews on this repo. Configure here.}

The fleet self-saturates: one SubagentStop burst spawned 15-30 processes that each re-parsed the same transcript, measured at 15.1s p50 wall idle against 1.55s for a single process, with per-hook durations escalating in launch order. dispatch.py now runs opted-in detectors in-process, sequentially, parsing the event once and passing the same parsed payload to every detector. A hook opts in with subagent_stop.dispatch in its manifest, naming a MODULE:FUNCTION entry point (default detect:detect). A detector with no resolvable entry point still runs as a subprocess and its metrics row is marked subprocess_fallback. Each detector keeps its own verdict, its own runs.jsonl row under its own hook name, and its own rule_ids; a detector that raises, exits, or overruns its 4s soft budget is caught and recorded without stopping its siblings. The event budget defaults to 9s for SubagentStop. mirror_stop_hooks_to_subagent_stop.py emits one dispatcher entry for the opted-in hooks and leaves every other hook's direct entry alone. (cherry picked from commit 531f5c8 on plan/hook-fleet-dispatcher-measure-the-herd-then-dispatch-it)

…op hooks dispatch through one process that parses the event once and shares one transcript parse across all opted-in detectors, emitting the same per-detector metrics rows and verdicts as the subprocess fleet. Review lane: behavior Safety invariant: Hook stdout and exit codes stay the same. This slice does not change the other four claims. Files: engine/hooks/_runner/dispatch.py Change types: bugfix Acceptance criteria: The branch contains only this claim, and the review step publishes one pull request. Exit code: 0 Invoker-Finalize-Id: aaa80940-f2d5-4d79-bb40-79c68fe333ad

…e5bede34 — Review claim: SubagentStop hooks dispatch through one process that parses the event once and shares one transcript parse across all opted-in detectors, emitting the same per-detector metrics rows and verdicts as the subprocess fleet. Review lane: behavior Safety invariant: Hook stdout and exit codes stay the same. This slice does not change the other four claims. Files: engine/hooks/_runner/dispatch.py Change types: bugfix Acceptance criteria: The branch contains only this claim, and the review step publishes one pull request.

Subprocess isolation was the hook fleet's free safety property, so the dispatcher has to prove it reproduced that property rather than assume it. _run_sdk_one caught a raising detector but left exit_code at 0, so outcome.classify saw a zero exit with a catstack-hook-error stderr line and recorded caught_error instead of crashed; the except branch now sets exit code 1. Four chaos fixtures under engine/hooks/_runner/tests/chaos (raise, sleep past the soft budget, exit 3, and a sibling that speaks) drive a dispatcher test asserting each failure gets its own row with the right outcome and exit code while both sibling detectors still emit their verdicts and rows. The fixtures are test-only: they live under tests/ and are registered in a scratch manifest, so the mirror script's name allowlist can never install them into a live settings.json. (cherry picked from commit b57f27c on plan/hook-fleet-dispatcher-measure-the-herd-then-dispatch-it)

…tectors that raise, hang past the soft budget, and exit non-zero each produce their own failure row while every sibling detector's verdict and row is still emitted. Review lane: behavior Safety invariant: Hook stdout and exit codes stay the same. This slice does not change the other four claims. Files: engine/hooks/_runner/tests/chaos Change types: bugfix Acceptance criteria: The branch contains only this claim, and the review step publishes one pull request. Exit code: 0 Invoker-Finalize-Id: 7ca2543e-d062-46fa-bbce-58264f6ae48f

…02624463 — Review claim: Fixture detectors that raise, hang past the soft budget, and exit non-zero each produce their own failure row while every sibling detector's verdict and row is still emitted. Review lane: behavior Safety invariant: Hook stdout and exit codes stay the same. This slice does not change the other four claims. Files: engine/hooks/_runner/tests/chaos Change types: bugfix Acceptance criteria: The branch contains only this claim, and the review step publishes one pull request.

cursor · 2026-10-11T16:26:02Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: bab474ae-5de3-4110-a837-2ae69bf4e237)

mergify · 2026-10-11T17:20:37Z

Queued — the merge queue status continues in this comment ↓.

Exit code: 0

cursor · 2026-10-11T19:23:26Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: bf916f3e-fdb5-45ac-9866-deb0f751d713)

The SubagentStop dispatcher records an in-process exception as exit 0. This pull request records that same exception as a crashed row, so a sibling can still speak while the failure stays visible. Co-authored-by: Cursor <cursoragent@cursor.com> Change-Id: If063e8ba2ce12336b8a515be81a05bb1885d2ba9

cursor · 2026-10-12T14:39:07Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 4027ed4e-2bdc-4ae1-9def-e71ea8591f04)

EdbertChan · 2026-10-12T15:52:15Z

@Mergifyio queue

mergify · 2026-10-12T15:52:33Z

Merge Queue Status

✅ Entered queue — 2026-10-12 15:52 UTC · Rule: admin-bypass · triggered by @EdbertChan with the @mergifyio queue command
❌ Checks timed out — 2026-10-12 18:22 UTC · on draft merge queue: checking #1333 + #1336 together on main (facdeaf) #1351
✂️ Bisecting to identify the failing PR (round 1/1)
🚫 Left the queue — 2026-10-12 20:24 UTC · at 92fb3ae898b1a0738f9cc44cbcee90247cf3bc44

This pull request spent 4 hours 31 minutes 37 seconds in the queue, including 1 hour 43 minutes 7 seconds running CI.

Reason

The merge conditions cannot be satisfied: the checks did not pass within the checks timeout of 1 hour

These checks were still running:

test

Failing checks:

🟠 test (job log)

Hint

You may have to fix your CI before adding the pull request to the queue again.

If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

Requeued — the merge queue status continues in this comment ↓.

EdbertChan · 2026-10-12T21:58:46Z

@Mergifyio queue

mergify · 2026-10-12T21:59:09Z

Merge Queue Status

✅ Entered queue — 2026-10-12 21:59 UTC · Rule: admin-bypass · triggered by @EdbertChan with the @mergifyio queue command
✅ Checks passed · on draft merge queue: checking #1358 + #1359 + #1336 together on main (76f7a53) #1366
✅ Merged — 2026-10-12 22:58 UTC · at 31c49a89d4a24cc352929e56250beeddf27e42c6 · squash

This pull request spent 59 minutes 33 seconds in the queue, including 51 minutes 41 seconds running CI.

Required conditions to merge

check-success = lint
check-success = test
check-success = validate

Invoker Bot and others added 6 commits October 11, 2026 15:34

EdbertChan added the admin-bypass label Oct 11, 2026

CI Bot added 2 commits October 11, 2026 19:22

Fix lint in chaos exit fixture

6bbfc6e

invoker: wf-1791746497228-5/repair — Repair PR #1336 (failed check lint)

1c10382

Exit code: 0

mergify Bot added the queued label Oct 12, 2026

mergify Bot added dequeued and removed queued labels Oct 12, 2026

mergify Bot added queued and removed dequeued labels Oct 12, 2026

mergify Bot mentioned this pull request Oct 12, 2026

merge queue: checking #1358 + #1359 + #1336 together on main (76f7a53) #1366

Closed

12 tasks done

mergify Bot merged commit 31c49a8 into main Oct 12, 2026
6 checks passed

mergify Bot removed the queued label Oct 12, 2026

Navigation Menu

Repository navigation

Hook fleet (5) chaos isolation fixtures - #1336

Hook fleet (5) chaos isolation fixtures#1336
mergify[bot] merged 9 commits into
mainfrom
plan/hook-fleet-5-chaos-isolation-fixtures

EdbertChan commented Oct 11, 2026 •

edited by cursor Bot

Loading

Uh oh!

cursor Bot commented Oct 11, 2026

Uh oh!

mergify Bot commented Oct 11, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Oct 11, 2026

Uh oh!

cursor Bot commented Oct 12, 2026

Uh oh!

EdbertChan commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

EdbertChan commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

EdbertChan commented Oct 11, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review Claim

Review Lane

Review Unit

Safety Invariant

Slice Rationale

Non-goals

Measured

Test Plan

Revert Plan

Uh oh!

cursor Bot commented Oct 11, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

mergify Bot commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Oct 11, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

cursor Bot commented Oct 12, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

EdbertChan commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

EdbertChan commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

EdbertChan commented Oct 11, 2026 •

edited by cursor Bot

Loading

mergify Bot commented Oct 11, 2026 •

edited

Loading

mergify Bot commented Oct 12, 2026 •

edited

Loading

mergify Bot commented Oct 12, 2026 •

edited

Loading