Hook fleet (2) replay harness by EdbertChan · Pull Request #1333 · EdbertChan/catstack

EdbertChan · 2026-10-11T15:04:08Z

Summary

This slice saves hook event inputs only when recording is enabled. It replays those inputs through two hook commands.

The report shows where the commands differ in output, context, findings, or exit status.

Review Claim

An opt-in recorder plus replay harness can compare installed hook commands against the same recorded events without changing hook output or exit status.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Hook stdout and exit codes remain unchanged. Recording is opt-in, bounded, and isolated from replay scratch state.

Slice Rationale

This keeps payload capture and replay together because the recorder supplies the exact inputs the comparison harness consumes.

The slice covers one claim from the failed dispatcher workflow and stays separate from the other four fleet claims.

Non-goals

No default payload recording.
No change to live hook dispatch or detector behavior.
No changes to the other four hook-fleet claims.
No user-interface changes.

Architecture

Before

graph TD
    A["hook event"] --> B["installed hook command"]
    B --> C["stdout and exit code"]

After

graph TD
    A["hook event"] --> B["optional bounded payload recording"]
    A --> C["installed hook command"]
    B --> D["recorded payload"]
    D --> E["fleet command and dispatcher command"]
    E --> F["verdict-equivalence report"]
    C --> G["unchanged stdout and exit code"]

Measured

none: this slice makes no performance or capacity claim beyond bounded payload retention.

Test Plan

python3 engine/skills/make-pr/scripts/preflight.py --base origin/main --commit-only --dry-run — unit engine-runtime: 5 file(s); ok preflight passed.
python3 scripts/ci/check_hook_test_coverage.py engine/hooks/_runner — check_hook_test_coverage: OK (1 hook(s) checked).
python3 -m unittest engine/hooks/_runner/tests/test_run.py scripts/test/test_replay_hook_payloads.py — Ran 25 tests; OK.
ruff check --select E9,F engine/hooks/_runner/run.py engine/hooks/_runner/tests/test_run.py scripts/test/replay_hook_payloads.py scripts/test/test_replay_hook_payloads.py — fails on unused import os in scripts/test/test_replay_hook_payloads.py:5.
pnpm run check:comments — unavailable: ERR_PNPM_NO_SCRIPT Missing script: check:comments.

Revert Plan

Safe to revert? Yes
Revert command: git revert <merge-commit-sha>
Post-revert steps: Remove any opt-in payload directory created for local replay.
Data migration? No

Note

Low Risk
Recording is off by default and bounded; the runner only adds a pre-subprocess write on the existing hook path, with tests asserting passthrough behavior.

Overview
Adds opt-in hook stdin capture in the metrics runner and a replay harness to compare two hook command implementations on the same events.

When CATSTACK_HOOK_PAYLOAD_DIR is set, run.py writes raw stdin bytes to <sha256-prefix>-<hook>.json immediately before starting the hook subprocess, with size/retention caps and no change to hook stdin, stdout, or exit code. Recording failures surface as an extra stderr line only.

New scripts/test/replay_hook_payloads.py feeds each recorded payload through paired --fleet and --dispatcher shell commands (matching labels), runs them in an isolated scratch env (no live metrics or recursive payload recording), and reports match/mismatch on stdout, context, block verdict, exit code, and findings—with optional JSON output.

^{Reviewed by Cursor Bugbot for commit 9787af8. Bugbot is set up for automated code reviews on this repo. Configure here.}

Review claim: - An opt-in payload recorder in the runner plus a replay script can feed recorded hook-event payloads through installed hook commands and produce a verdict-equivalence report (stdout/context/block/exit) per detector. Review lane: behavior Safety invariant: - Recording is off unless CATSTACK_HOOK_PAYLOAD_DIR is set; when on it never alters hook stdin/stdout/exit and caps each payload (default 256KB) and total dir (default 2000 files, oldest evicted). - Replay runs against a scratch metrics/state dir, never live settings or the live runs.jsonl. Goal: - Runner: when env var set, append the raw stdin payload to $CATSTACK_HOOK_PAYLOAD_DIR/<payload_hash>-<hook>.json before spawning the hook child, where <payload_hash> is sha256(stdin)[:12]. - scripts/test/replay_hook_payloads.py: for each recorded payload, run the hook command under a scratch env, capture stdout/stderr/exit/ findings; emit an equivalence report JSON + table comparing two command sets (fleet vs dispatcher args taken as parameters). Motivation: - The catastrophic regression risk of the dispatcher is a detector that silently stops speaking; the only credible gate is byte-identical verdicts on real payloads. Cherry-picked from origin/plan/hook-fleet-dispatcher-measure-the-herd-then-dispatch-it (e356e8c), scoped to only this claim's files. Dropped this slice's dependency on the separate metrics-event-uid claim (17ef151, not on this branch): the payload recorder computes its own local hash instead of reusing a published `event_uid` metrics-row field, and the README wording and test imports were adjusted to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

… payload recorder in the runner plus a replay script can feed recorded hook-event payloads through installed hook commands and produce a verdict-equivalence report per detector. Review lane: behavior Safety invariant: Hook stdout and exit codes stay the same. This slice does not change the other four claims. Files: engine/hooks/_runner/run.py, scripts/test/replay_hook_payloads.py Change types: bugfix Acceptance criteria: The branch contains only this claim, and the review step publishes one pull request. Exit code: 0 Invoker-Finalize-Id: 61e213a0-a6cd-4950-ba23-0acc17742997

…-1bc38490 — Review claim: An opt-in payload recorder in the runner plus a replay script can feed recorded hook-event payloads through installed hook commands and produce a verdict-equivalence report per detector. Review lane: behavior Safety invariant: Hook stdout and exit codes stay the same. This slice does not change the other four claims. Files: engine/hooks/_runner/run.py, scripts/test/replay_hook_payloads.py Change types: bugfix Acceptance criteria: The branch contains only this claim, and the review step publishes one pull request.

cursor · 2026-10-11T15:04:16Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 5a6ef6ca-09c9-4fb5-b62c-6a9f5611c6e2)

mergify · 2026-10-11T19:10:58Z

Queued — the merge queue status continues in this comment ↓.

Exit code: 0

cursor · 2026-10-11T19:22:49Z

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 563c84c2-5864-4bf7-bf87-610764c2565d)

EdbertChan · 2026-10-12T14:36:23Z

@Mergifyio queue

mergify · 2026-10-12T14:36:28Z

Merge Queue Status

✅ Entered queue — 2026-10-12 14:36 UTC · Rule: admin-bypass · triggered by @EdbertChan with the @mergifyio queue command
❌ Checks timed out · on draft merge queue: checking #1333 on main (73f06fb), stacked on #1323 #1341
🚫 Left the queue — 2026-10-12 15:37 UTC · at 9787af8f07826c77ba0072030a241cf253e6a08c

This pull request spent 1 hour 1 minute 3 seconds in the queue, including 1 hour 31 seconds running CI.

Reason

The merge conditions cannot be satisfied: the checks did not pass within the checks timeout of 1 hour

These checks were still running:

test

Failing checks:

🟠 test (job log)

Hint

You may have to fix your CI before adding the pull request to the queue again.

If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

Waiting for

check-success = test

All conditions

check-success = test
check-success = lint
check-success = validate

Requeued — the merge queue status continues in this comment ↓.

EdbertChan · 2026-10-12T15:52:10Z

@Mergifyio queue

mergify · 2026-10-12T15:52:16Z

Merge Queue Status

✅ Entered queue — 2026-10-12 15:52 UTC · Rule: admin-bypass · triggered by @EdbertChan with the @mergifyio queue command
❌ Checks timed out — 2026-10-12 18:22 UTC · on draft merge queue: checking #1333 + #1336 together on main (facdeaf) #1351
✂️ Bisecting to identify the failing PR (round 1/1)
✅ Checks passed · on draft merge queue: checking #1333 on main (facdeaf) #1353
✅ Merged — 2026-10-12 20:24 UTC · at 2f93d6cd21f9768359faa56cbcc7b87698863376 · squash

This pull request spent 4 hours 31 minutes 51 seconds in the queue, including 2 hours 43 minutes 37 seconds running CI.

Required conditions to merge

check-success = lint
check-success = test
check-success = validate

EdbertChan and others added 3 commits October 11, 2026 14:53

EdbertChan added the admin-bypass label Oct 11, 2026

CI Bot added 2 commits October 11, 2026 19:22

Fix replay hook payload lint

d63b1fc

invoker: wf-1791746491297-4/repair — Repair PR #1333 (failed check lint)

9787af8

Exit code: 0

mergify Bot added the queued label Oct 12, 2026

mergify Bot added dequeued and removed queued labels Oct 12, 2026

mergify Bot added the queued label Oct 12, 2026

mergify Bot mentioned this pull request Oct 12, 2026

merge queue: checking #1333 on main (2745ccd), stacked on #1332, #1314 and #1309 #1346

Closed

5 of 6 tasks

mergify Bot removed the dequeued label Oct 12, 2026

mergify Bot merged commit 2f93d6c into main Oct 12, 2026
6 checks passed

mergify Bot removed the queued label Oct 12, 2026

Navigation Menu

Repository navigation

Hook fleet (2) replay harness - #1333

Hook fleet (2) replay harness#1333
mergify[bot] merged 5 commits into
mainfrom
plan/hook-fleet-2-replay-harness

EdbertChan commented Oct 11, 2026 •

edited by cursor Bot

Loading

Uh oh!

cursor Bot commented Oct 11, 2026

Uh oh!

mergify Bot commented Oct 11, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Oct 11, 2026

Uh oh!

EdbertChan commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

EdbertChan commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

EdbertChan commented Oct 11, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review Claim

Review Lane

Review Unit

Safety Invariant

Slice Rationale

Non-goals

Architecture

Before

After

Measured

Test Plan

Revert Plan

Uh oh!

cursor Bot commented Oct 11, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

mergify Bot commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Oct 11, 2026

Bugbot couldn't run - usage limit reached

Uh oh!

EdbertChan commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Reason

Hint

Uh oh!

EdbertChan commented Oct 12, 2026

Uh oh!

mergify Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Queue Status

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

EdbertChan commented Oct 11, 2026 •

edited by cursor Bot

Loading

mergify Bot commented Oct 11, 2026 •

edited

Loading

mergify Bot commented Oct 12, 2026 •

edited

Loading

mergify Bot commented Oct 12, 2026 •

edited

Loading