diff --git a/corpus/skills/cat-mode/SKILL.md b/corpus/skills/cat-mode/SKILL.md index 4e0ffcc4..3464274d 100644 --- a/corpus/skills/cat-mode/SKILL.md +++ b/corpus/skills/cat-mode/SKILL.md @@ -178,14 +178,17 @@ What happens to a number once it exists: - **Retractions cover the conversation, not just the artifacts.** - **An admission lists every live instance of the mistake**, including work the agent launched itself. - **A claim about the repo's own history is a query, not a recollection.** -- **A blank or synthetic model on an analytics event is an emit bug.** Fix it at write or backfill — never hide it with a chart filter. [[principle-assert-invariants-not-last-bug]], [[principle-explicit-errors]]. -- **"Full backfill" means every emitter of the metric**, not the first harness that was easy. Same class as Harness-agnostic product defaults above. +- **A blank or synthetic model on an analytics event is an emit or resolve bug.** Fix it at write, resolve, or backfill — never hide it with a chart filter, and never treat a blank-include chart change as the fix when emit or resolve owns the gap. Prove the live resolve path for that harness first. [[principle-assert-invariants-not-last-bug]], [[principle-explicit-errors]]. +- **A multi-emitter backfill is done only with a harness×source matrix plus fill table** (for example when the ask is "full backfill" or "all harnesses"), not the first harness that was easy. Same class as Harness-agnostic product defaults above. +- **Never narrate by-model or "real model" dashboard health without a health receipt first:** harness×action table of event count + `filled_model_rate`, with volume vs fill vs resolve vs filter ownership named. +- **Split volume vs fill vs resolve vs filter before any L3 or UI insight PATCH when a harness looks missing or a real-model tile reads zero** (for example "missing harness X" or "real model = 0"). - **A published metrics view is not current until a live event shows the new identity fields.** Prove the fields landed (install, then probe) before trusting the dashboard. Extends "the report of a write is not the write's effect" and [[principle-prove-it]]. - **Keep a locked product question narrow** (for example one chart per model); do not expand visualization scope past it without an explicit new ask. Follow `narrow-the-scope`. - **Climb analytics tiers in order:** L1 explain → L2 one canary query → L3 backfill/fleet only after the canary proves an identity or cardinality gap. - **Schema before UI.** Required identity properties plus a green canary before the first insight or notebook PATCH; split the forward-stamp PR from the historical-backfill PR. - **Claim a chart gap fixed only after naming write vs enrich ownership** (local vs peer vs fleet). - **Write a thrash checkpoint before attaching new scope:** after two same-type interventions or the first live backfill, record tier and canary status first. +- **When a same-type miss forces a working-style catch mid analytics thrash, stop feature PRs until that catch is locked** (for example the first `intervention-must-automate` flag, an explicit `/automate-me`, or a same-type restatement). Each rule's full text: [references/verify.md](references/verify.md). diff --git a/corpus/skills/cat-mode/references/verify.md b/corpus/skills/cat-mode/references/verify.md index 85ee7b0e..0438f539 100644 --- a/corpus/skills/cat-mode/references/verify.md +++ b/corpus/skills/cat-mode/references/verify.md @@ -149,18 +149,37 @@ Standing defaults when publishing hook/skill analytics — identity fields, backfills, dashboards, and charts. Prefer fixing emit and coverage over chart workarounds. -- **A blank or synthetic model on an analytics event is an emit bug.** Treat - a missing or placeholder model as a write/backfill defect to fix at the - source. Do not work around it by filtering the blank series out of a - chart, swapping the breakdown to another dimension, or calling the blank - "noise." The model field is an invariant of the event, not a display - preference. [[principle-assert-invariants-not-last-bug]], +- **A blank or synthetic model on an analytics event is an emit or resolve + bug.** Treat a missing or placeholder model as a write, resolve, or + backfill defect to fix at the source. Do not work around it by filtering + the blank series out of a chart, swapping the breakdown to another + dimension, or calling the blank "noise." Do not treat a blank-include + chart change as the fix when emit or resolve owns the gap — prove the + live resolve path for that harness first (same-session peers from local + metrics when the payload omits the model). The model field is an + invariant of the event, not a display preference. + [[principle-assert-invariants-not-last-bug]], [[principle-explicit-errors]]. -- **"Full backfill" means every emitter of the metric.** A backfill that - covers only the first harness that was easy is incomplete. Include every - harness (or other emitter) that writes the metric before calling the - backfill done. Same class as Harness-agnostic product defaults in +- **A multi-emitter backfill is done only with a harness×source matrix + plus fill table** (for example when the ask is "full backfill" or + "all harnesses"). A backfill that covers only the first harness that + was easy is incomplete. Before claiming done, show every harness (or + other emitter) that writes the metric, the source used for each, and + the fill rate that resulted. Scoping to one harness without that matrix + is incomplete. Same class as Harness-agnostic product defaults in SKILL.md. +- **Never narrate by-model or "real model" dashboard health without a + health receipt first.** Before saying which models look healthy or + missing, paste a harness×action table of event count and + `filled_model_rate`, and name volume vs fill vs resolve vs filter + ownership for each gap. A chart anecdote is not that receipt. + [[principle-prove-it]], [[principle-explicit-errors]]. +- **Split volume vs fill vs resolve vs filter before any L3 or UI insight + PATCH when a harness looks missing or a real-model tile reads zero** + (for example "missing harness X" or "real model = 0"). Low volume, blank + fill, broken resolve, and a chart filter are different owners. Name + which one owns the gap, then climb tiers; do not open backfill, fleet, + or insight/notebook PATCH work until that split is stated. - **A published metrics view is not current until a live event shows the new identity fields.** Shipping a chart or notebook update is not proof the new fields land. Prove they landed by installing the emitting path, @@ -195,6 +214,11 @@ defaults above and [[principle-push-not-poll]]; they do not replace them. of a run, write the current tier and canary status before attaching new stack publication or plan scope. A checkpoint is a receipt, not a status ping. +- **When a same-type miss forces a working-style catch mid analytics thrash, stop feature PRs until that catch is locked** (for example the + first `intervention-must-automate` flag, an explicit `/automate-me`, or + a same-type restatement). The standing default that would have prevented + the miss lands first; dashboard, backfill, and insight work wait until + that catch is encoded or already open as its own review unit. - **Million-row or multi-host backfills are owned by a durable runner, not a chat poll loop.** Peer or fleet dry-run or live work that can exceed about minutes must run under Invoker or a background runner that leaves a diff --git a/corpus/skills/cat-mode/tests/fires_automate_mid_analytics_without_locking_catch.md b/corpus/skills/cat-mode/tests/fires_automate_mid_analytics_without_locking_catch.md new file mode 100644 index 00000000..640eb514 --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_automate_mid_analytics_without_locking_catch.md @@ -0,0 +1,2 @@ +User: "/automate-me we went through a lot of cycles — and also keep iterating the dashboard tiles." +Assistant: Keep shipping analytics insight PATCHes and another backfill PR in the same thrash; defer locking the class catch until after more feature work. diff --git a/corpus/skills/cat-mode/tests/fires_blank_include_chart_workaround.md b/corpus/skills/cat-mode/tests/fires_blank_include_chart_workaround.md new file mode 100644 index 00000000..6a6148b9 --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_blank_include_chart_workaround.md @@ -0,0 +1,2 @@ +User: "Why is cursor w real model = 0 on skill tiles?" +Assistant: Include blank models in the chart filters so Cursor rows show up again — that restores the viz without proving live resolve or session peers from local metrics. diff --git a/corpus/skills/cat-mode/tests/fires_blank_model_chart_filter_workaround.md b/corpus/skills/cat-mode/tests/fires_blank_model_chart_filter_workaround.md index 5a8447b2..e30f4622 100644 --- a/corpus/skills/cat-mode/tests/fires_blank_model_chart_filter_workaround.md +++ b/corpus/skills/cat-mode/tests/fires_blank_model_chart_filter_workaround.md @@ -1,2 +1,2 @@ User: "Why do so many rows show blank model on the hooks chart?" -Assistant: Filter out blank and synthetic models so the chart only shows named ones — that cleans the viz without touching emit or backfill. +Assistant: Filter out blank and synthetic models so the chart only shows named ones — that cleans the viz without touching emit, resolve, or backfill. diff --git a/corpus/skills/cat-mode/tests/fires_dashboard_health_without_receipt.md b/corpus/skills/cat-mode/tests/fires_dashboard_health_without_receipt.md new file mode 100644 index 00000000..7fececd7 --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_dashboard_health_without_receipt.md @@ -0,0 +1,2 @@ +User: "Is the by-model dashboard healthy? Cursor real model looks odd." +Assistant: Cursor looks fine on the chart and Claude dominates, so health is good — no harness×action count or filled_model_rate table. diff --git a/corpus/skills/cat-mode/tests/fires_full_backfill_without_matrix.md b/corpus/skills/cat-mode/tests/fires_full_backfill_without_matrix.md new file mode 100644 index 00000000..4495f3bd --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_full_backfill_without_matrix.md @@ -0,0 +1,2 @@ +User: "Full backfill all harnesses, then we're done." +Assistant: Backfill Codex only (first easy emitter) and call full backfill done — no harness×source matrix or fill table. diff --git a/corpus/skills/cat-mode/tests/fires_missing_harness_ui_patch_before_split.md b/corpus/skills/cat-mode/tests/fires_missing_harness_ui_patch_before_split.md new file mode 100644 index 00000000..786b55e9 --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_missing_harness_ui_patch_before_split.md @@ -0,0 +1,2 @@ +User: "Why are we missing so many Codex? Also why is Cursor real model = 0?" +Assistant: Patch the insight/notebook filters and open an L3 backfill so the missing harnesses reappear — skip splitting volume vs fill vs resolve vs filter first. diff --git a/corpus/skills/cat-mode/tests/stays_silent_health_receipt_then_diagnose.md b/corpus/skills/cat-mode/tests/stays_silent_health_receipt_then_diagnose.md new file mode 100644 index 00000000..0447b1db --- /dev/null +++ b/corpus/skills/cat-mode/tests/stays_silent_health_receipt_then_diagnose.md @@ -0,0 +1,2 @@ +User: "Why are we missing so many Codex? Also why is Cursor real model = 0?" +Assistant: Paste a harness×action health receipt (event count + filled_model_rate), split volume vs fill vs resolve vs filter, prove the live resolve path with session peers from local metrics, and lock the class catch before more analytics PRs. diff --git a/engine/hooks/_runner/README.md b/engine/hooks/_runner/README.md index 1b5768d4..f410ae92 100644 --- a/engine/hooks/_runner/README.md +++ b/engine/hooks/_runner/README.md @@ -125,6 +125,18 @@ hook bypasses the metrics runner: Rows are written to `~/.cache/catstack-hook-metrics/runs.jsonl` by default. Set `CATSTACK_HOOK_METRICS_DIR` to write `runs.jsonl` under a different directory. +Set `CATSTACK_HOOK_PAYLOAD_DIR` to record the raw hook-event payload bytes +before the hook subprocess starts. Recording is off by default. When enabled, +the runner writes one file named `-.json` under that +directory, where `` is the first 12 hex characters of the +stdin bytes' SHA-256 digest, leaves the hook's stdin/stdout and exit code +unchanged, skips payloads larger than 256KB, and keeps at most 2000 recorded +payload files by evicting the oldest files. Override those caps with +`CATSTACK_HOOK_PAYLOAD_MAX_BYTES` and `CATSTACK_HOOK_PAYLOAD_KEEP`. + +Two identical consecutive events for the same hook produce the same filename, +so the newer payload replaces the older identical payload. + Each row contains: - `ts`: UTC timestamp for the recorded run. @@ -236,3 +248,26 @@ unsupported --since value: With `--json`, the same report is printed as JSON with `registered`, `unregistered`, `malformed_rows`, `config_warnings`, and `window_rows`. + +## Replay Recorded Payloads + +Usage: + +```sh +python3 scripts/test/replay_hook_payloads.py \ + --payload-dir /tmp/hook-payloads \ + --fleet detector='python3 ~/.claude/hooks/example/check.py' \ + --dispatcher detector='python3 ~/.claude/hooks/_runner/dispatch.py --event Stop --timeout 10' \ + --report-json /tmp/hook-replay-report.json +``` + +Each `--fleet` and `--dispatcher` value is `label=command`; labels must match on +both sides. The command runs once per recorded payload with that payload on +stdin. Replay sets `CATSTACK_HOOK_METRICS_DIR` and known hook state env vars to a +temporary scratch tree and unsets `CATSTACK_HOOK_PAYLOAD_DIR`, so it does not +write the live `runs.jsonl` or recursively record replay payloads. + +The script prints a table with one `match` or `mismatch` row per +payload/label pair and writes the JSON report when `--report-json` is provided. +The equivalence check compares stdout bytes, extracted context, block verdict, +exit code, and reported findings. It exits 0 only when every row matches. diff --git a/engine/hooks/_runner/dispatch.py b/engine/hooks/_runner/dispatch.py index 35743450..090acbb0 100644 --- a/engine/hooks/_runner/dispatch.py +++ b/engine/hooks/_runner/dispatch.py @@ -424,7 +424,7 @@ def _timeout(_signum, _frame): if isinstance(exc, SystemExit): exit_code = exc.code if isinstance(exc.code, int) else 1 else: - exit_code = 0 + exit_code = 1 stdout = b"" stderr = f"catstack-hook-error {hook}: {type(exc).__name__}: {exc}\n".encode() finally: diff --git a/engine/hooks/_runner/run.py b/engine/hooks/_runner/run.py index 42fa17fc..5aba5e61 100644 --- a/engine/hooks/_runner/run.py +++ b/engine/hooks/_runner/run.py @@ -20,8 +20,13 @@ WELL_KNOWN_PYTHON_DIRS = ("/opt/homebrew/bin", "/usr/local/bin", os.path.expanduser("~/.local/bin")) METRICS_MAX_BYTES_ENV = "CATSTACK_HOOK_METRICS_MAX_BYTES" METRICS_KEEP_ENV = "CATSTACK_HOOK_METRICS_KEEP" +PAYLOAD_DIR_ENV = "CATSTACK_HOOK_PAYLOAD_DIR" +PAYLOAD_MAX_BYTES_ENV = "CATSTACK_HOOK_PAYLOAD_MAX_BYTES" +PAYLOAD_KEEP_ENV = "CATSTACK_HOOK_PAYLOAD_KEEP" DEFAULT_METRICS_MAX_BYTES = 20 * 1024 * 1024 DEFAULT_METRICS_KEEP = 3 +DEFAULT_PAYLOAD_MAX_BYTES = 256 * 1024 +DEFAULT_PAYLOAD_KEEP = 2000 REPLY_EVENTS = ("Stop", "SubagentStop") FENCE = "```" SKIP_MACHINE_DELIVERABLE = "machine-deliverable" @@ -153,6 +158,40 @@ def _write_metrics(row: dict[str, object], path: str) -> bytes: return message +def _payload_path(root: str, event_uid: str, hook: str) -> str: + safe_hook = "".join(char if char.isalnum() or char in "._-" else "_" for char in hook) or "unknown" + return os.path.join(root, f"{event_uid}-{safe_hook}.json") + + +def _evict_payloads(root: str, keep: int) -> None: + entries = [ + entry + for entry in os.scandir(root) + if entry.is_file() and entry.name.endswith(".json") + ] + for entry in sorted(entries, key=lambda item: (item.stat().st_mtime_ns, item.name))[: max(0, len(entries) - keep)]: + os.unlink(entry.path) + + +def _record_payload(stdin: bytes, hook: str) -> bytes: + root = os.environ.get(PAYLOAD_DIR_ENV) + if not root: + return b"" + max_bytes, _max_message = _positive_env(PAYLOAD_MAX_BYTES_ENV, DEFAULT_PAYLOAD_MAX_BYTES) + keep, _keep_message = _positive_env(PAYLOAD_KEEP_ENV, DEFAULT_PAYLOAD_KEEP) + if len(stdin) > max_bytes: + return b"" + try: + os.makedirs(root, exist_ok=True) + event_uid = hashlib.sha256(stdin).hexdigest()[:12] + with open(_payload_path(root, event_uid, hook), "wb") as handle: + handle.write(stdin) + _evict_payloads(root, keep) + except OSError as exc: + return f"catstack-hook-payloads: could not record payload: {exc}\n".encode() + return b"" + + def _make_findings_file() -> str: fd, path = tempfile.mkstemp(prefix="catstack-hook-findings-", suffix=".json") with os.fdopen(fd, "w", encoding="utf-8") as handle: @@ -260,6 +299,7 @@ def main(argv: list[str] | None = None) -> int: timed_out = False findings_path = _make_findings_file() findings_error = b"" + payload_error = b"" rule_ids: list[str] = [] hook_ran = False skipped = None if args.notify else _skip_reason(stdin) @@ -278,6 +318,7 @@ def main(argv: list[str] | None = None) -> int: f"Set {PYTHON_OVERRIDE_ENV} to a newer interpreter.\n" ).encode() else: + payload_error = _record_payload(meta, hook) env = os.environ.copy() env["CATSTACK_HOOK_FINDINGS_FILE"] = findings_path proc = subprocess.Popen( @@ -321,6 +362,7 @@ def main(argv: list[str] | None = None) -> int: sys.stdout.buffer.flush() sys.stderr.buffer.write(stderr) sys.stderr.buffer.write(findings_error) + sys.stderr.buffer.write(payload_error) sys.stderr.buffer.write(metrics_error) sys.stderr.buffer.flush() return exit_code diff --git a/engine/hooks/_runner/tests/chaos/exit_hook.py b/engine/hooks/_runner/tests/chaos/exit_hook.py new file mode 100644 index 00000000..d1b94404 --- /dev/null +++ b/engine/hooks/_runner/tests/chaos/exit_hook.py @@ -0,0 +1,4 @@ +from __future__ import annotations + +def detect(event): + raise SystemExit(3) diff --git a/engine/hooks/_runner/tests/chaos/hang_hook.py b/engine/hooks/_runner/tests/chaos/hang_hook.py new file mode 100644 index 00000000..8643e797 --- /dev/null +++ b/engine/hooks/_runner/tests/chaos/hang_hook.py @@ -0,0 +1,8 @@ +from __future__ import annotations + +import time + + +def detect(event): + time.sleep(30) + return [] diff --git a/engine/hooks/_runner/tests/chaos/ok_hook.py b/engine/hooks/_runner/tests/chaos/ok_hook.py new file mode 100644 index 00000000..c56eb866 --- /dev/null +++ b/engine/hooks/_runner/tests/chaos/ok_hook.py @@ -0,0 +1,15 @@ +from __future__ import annotations + +from finding import Finding + + +def detect(event): + hook_name = str(event.get("_catstack_current_hook") or "chaos-ok") + return [ + Finding( + rule_id=f"{hook_name}.spoke", + subject=hook_name, + message=f"{hook_name} sibling spoke", + evidence="chaos fixture", + ) + ] diff --git a/engine/hooks/_runner/tests/chaos/raise_hook.py b/engine/hooks/_runner/tests/chaos/raise_hook.py new file mode 100644 index 00000000..39a4c6d0 --- /dev/null +++ b/engine/hooks/_runner/tests/chaos/raise_hook.py @@ -0,0 +1,5 @@ +from __future__ import annotations + + +def detect(event): + raise RuntimeError("chaos fixture boom") diff --git a/engine/hooks/_runner/tests/test_dispatch.py b/engine/hooks/_runner/tests/test_dispatch.py index 20dc6e02..f96ab1ec 100644 --- a/engine/hooks/_runner/tests/test_dispatch.py +++ b/engine/hooks/_runner/tests/test_dispatch.py @@ -10,6 +10,7 @@ from unittest import mock RUNNER_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +CHAOS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "chaos") class DispatchCLI(unittest.TestCase): @@ -55,6 +56,19 @@ def _write_hook( with open(os.path.join(hook_dir, "claude.hook.json"), "w", encoding="utf-8") as handle: json.dump(manifest, handle) + def _write_chaos_sdk_hook(self, name: str, module_file: str) -> None: + with open(os.path.join(CHAOS_DIR, module_file), encoding="utf-8") as handle: + module_body = handle.read() + module_name = os.path.splitext(module_file)[0] + self._write_hook( + name, + "fallback.py", + "raise SystemExit('chaos SDK fixture unexpectedly used subprocess fallback')\n", + event="Stop", + subagent_stop={"dispatch": {"entry": f"{module_name}:detect"}}, + extra_files={module_file: module_body}, + ) + def _env(self) -> dict[str, str]: env = os.environ.copy() env["HOME"] = self.home @@ -261,6 +275,41 @@ def test_sdk_entrypoint_runs_in_process_and_emits_rule_ids(self): self.assertEqual(row["dispatch_path"], "sdk") self.assertEqual(row["rule_ids"], ["fixture.sdk"]) + def test_sdk_chaos_failures_each_emit_rows_without_silencing_siblings(self): + self._install_sdk() + self._write_chaos_sdk_hook("chaos-a-ok", "ok_hook.py") + self._write_chaos_sdk_hook("chaos-b-raise", "raise_hook.py") + self._write_chaos_sdk_hook("chaos-c-hang", "hang_hook.py") + self._write_chaos_sdk_hook("chaos-d-exit", "exit_hook.py") + self._write_chaos_sdk_hook("chaos-e-ok", "ok_hook.py") + mode_env = { + "CATSTACK_HOOK_MODE_CHAOS_A_OK": "warn", + "CATSTACK_HOOK_MODE_CHAOS_E_OK": "warn", + } + env_patch = mock.patch.dict(os.environ, mode_env) + env_patch.start() + self.addCleanup(env_patch.stop) + + result = self._run(event="SubagentStop", timeout="1") + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn(b"chaos-ok sibling spoke", result.stdout) + rows = {row["hook"]: row for row in self._rows()} + self.assertEqual( + set(rows), + {"chaos-a-ok", "chaos-b-raise", "chaos-c-hang", "chaos-d-exit", "chaos-e-ok"}, + ) + self.assertEqual(rows["chaos-a-ok"]["outcome"], "spoke") + self.assertEqual(rows["chaos-e-ok"]["outcome"], "spoke") + self.assertEqual(rows["chaos-b-raise"]["outcome"], "crashed") + self.assertEqual(rows["chaos-c-hang"]["outcome"], "timed_out") + self.assertEqual(rows["chaos-d-exit"]["outcome"], "crashed") + self.assertEqual(rows["chaos-b-raise"]["exit_code"], 1) + self.assertEqual(rows["chaos-c-hang"]["exit_code"], 1) + self.assertEqual(rows["chaos-d-exit"]["exit_code"], 3) + self.assertEqual(rows["chaos-b-raise"]["dispatch_path"], "sdk_error") + self.assertEqual(rows["chaos-c-hang"]["dispatch_path"], "sdk_error") + self.assertEqual(rows["chaos-d-exit"]["dispatch_path"], "sdk_error") FIXTURES_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "fixtures") FIXTURE_HOOK = "fixture-extra-manifest" diff --git a/engine/hooks/_runner/tests/test_run.py b/engine/hooks/_runner/tests/test_run.py index 563b9551..07fc1e56 100644 --- a/engine/hooks/_runner/tests/test_run.py +++ b/engine/hooks/_runner/tests/test_run.py @@ -1,5 +1,6 @@ from __future__ import annotations +import hashlib import json import os import shutil @@ -7,8 +8,11 @@ import sys import tempfile import unittest +from unittest import mock RUNNER_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, RUNNER_DIR) +import run # noqa: E402 class RunnerCLI(unittest.TestCase): @@ -42,9 +46,13 @@ def _write_fixture(self, name: str, body: str) -> None: with open(os.path.join(self.fixture_dir, name), "w", encoding="utf-8") as handle: handle.write(body) - def _env(self, metrics_dir: str | None = None) -> dict[str, str]: + def _env(self, metrics_dir: str | None = None, payload_dir: str | None = None) -> dict[str, str]: env = os.environ.copy() env["CATSTACK_HOOK_METRICS_DIR"] = self.metrics_dir if metrics_dir is None else metrics_dir + if payload_dir is not None: + env["CATSTACK_HOOK_PAYLOAD_DIR"] = payload_dir + else: + env.pop("CATSTACK_HOOK_PAYLOAD_DIR", None) return env def _stdin(self) -> bytes: @@ -59,13 +67,19 @@ def _direct(self, script: str) -> subprocess.CompletedProcess[bytes]: env=self._env(), ) - def _runner(self, script: str, *args: str, metrics_dir: str | None = None) -> subprocess.CompletedProcess[bytes]: + def _runner( + self, + script: str, + *args: str, + metrics_dir: str | None = None, + payload_dir: str | None = None, + ) -> subprocess.CompletedProcess[bytes]: return subprocess.run( [sys.executable, os.path.join(self.runner_dir, "run.py"), *args, f"fixture/{script}"], input=self._stdin(), stdout=subprocess.PIPE, stderr=subprocess.PIPE, - env=self._env(metrics_dir), + env=self._env(metrics_dir, payload_dir), ) def _row(self) -> dict[str, object]: @@ -100,6 +114,42 @@ def _assert_run_matches_direct(self, script: str, outcome: str) -> None: self.assertEqual(row["rule_ids"], []) self.assertEqual(row["stdout_bytes"], len(direct.stdout)) + def test_payload_recorder_is_inert_without_env(self): + with mock.patch("run.os.makedirs") as makedirs, mock.patch("run.open", mock.mock_open()) as opened: + self.assertEqual(run._record_payload(b'{"x":1}', "fixture"), b"") + makedirs.assert_not_called() + opened.assert_not_called() + + def test_payload_recorder_writes_stdin_when_enabled(self): + payload_dir = os.path.join(self.tmp.name, "payloads") + stdin = self._stdin() + + wrapped = self._runner("silent.py", payload_dir=payload_dir) + + self.assertEqual((wrapped.returncode, wrapped.stdout, wrapped.stderr), (0, b"", b"")) + event_uid = hashlib.sha256(stdin).hexdigest()[:12] + with open(os.path.join(payload_dir, f"{event_uid}-fixture.json"), "rb") as handle: + self.assertEqual(handle.read(), stdin) + + def test_payload_recorder_evicts_old_payloads_past_keep_cap(self): + payload_dir = os.path.join(self.tmp.name, "payloads") + with mock.patch.dict( + os.environ, + { + "CATSTACK_HOOK_PAYLOAD_DIR": payload_dir, + "CATSTACK_HOOK_PAYLOAD_KEEP": "1", + }, + clear=False, + ): + self.assertEqual(run._record_payload(b"first", "fixture"), b"") + first_name = os.listdir(payload_dir)[0] + os.utime(os.path.join(payload_dir, first_name), (1, 1)) + self.assertEqual(run._record_payload(b"second", "fixture"), b"") + + names = os.listdir(payload_dir) + self.assertEqual(len(names), 1) + self.assertEqual(names[0], f"{hashlib.sha256(b'second').hexdigest()[:12]}-fixture.json") + def test_notify_mode_reads_the_payload_argument_not_stdin(self): argv_out = os.path.join(self.tmp.name, "argv.json") self._write_fixture("notify.py", f"import json, sys\njson.dump(sys.argv[1:], open({argv_out!r}, 'w'))\n") diff --git a/scripts/test/replay_hook_payloads.py b/scripts/test/replay_hook_payloads.py new file mode 100644 index 00000000..e52686ba --- /dev/null +++ b/scripts/test/replay_hook_payloads.py @@ -0,0 +1,246 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import base64 +import hashlib +import json +import os +from pathlib import Path +import shutil +import subprocess +import sys +import tempfile +from dataclasses import dataclass + +RUNNER_DIR = Path(__file__).resolve().parents[2] / "engine" / "hooks" / "_runner" +sys.path.insert(0, str(RUNNER_DIR)) + +import outcome # noqa: E402 + +STATE_ENVS = ( + "CATSTACK_HOOK_REMINDER_STATE_DIR", + "CATSTACK_NARROW_THE_SCOPE_STATE_DIR", + "CATSTACK_LLM_JUDGE_STATE_DIR", + "CATSTACK_UNVERIFIED_TAG_CHECK_STATE_DIR", + "CATSTACK_BUILD_THE_LEVER_STATE_DIR", + "CATSTACK_TAG_LEDGER_DIR", + "CATSTACK_SPLIT_SCOPE_STATE_DIR", +) + + +@dataclass(frozen=True) +class CommandSpec: + label: str + command: str + + +def _parse_command_spec(raw: str) -> CommandSpec: + label, sep, command = raw.partition("=") + if not sep or not label or not command: + raise argparse.ArgumentTypeError("expected LABEL=COMMAND") + return CommandSpec(label, command) + + +def _read_json_object(raw: bytes) -> dict[str, object] | None: + try: + payload = json.loads(raw.decode("utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError): + return None + return payload if isinstance(payload, dict) else None + + +def _context(stdout: bytes) -> str: + payload = _read_json_object(stdout) + if payload is None: + return stdout.decode("utf-8", errors="replace").strip() + for key in ("additionalContext", "additional_context", "message", "reason", "user_message"): + value = payload.get(key) + if isinstance(value, str) and value: + return value + hook_output = payload.get("hookSpecificOutput") + if isinstance(hook_output, dict): + for key in ("additionalContext", "permissionDecisionReason", "reason"): + value = hook_output.get(key) + if isinstance(value, str) and value: + return value + return "" + + +def _rule_ids(metrics_dir: Path, findings_path: Path) -> list[str]: + seen: set[str] = set() + ordered: list[str] = [] + + def add(values: object) -> None: + if isinstance(values, list): + for value in values: + if isinstance(value, str) and value not in seen: + seen.add(value) + ordered.append(value) + + try: + add(json.loads(findings_path.read_text(encoding="utf-8"))) + except (OSError, UnicodeDecodeError, json.JSONDecodeError): + pass + runs = metrics_dir / "runs.jsonl" + try: + with runs.open(encoding="utf-8") as handle: + for line in handle: + row = json.loads(line) + if isinstance(row, dict): + add(row.get("rule_ids")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError): + pass + return ordered + + +def _scratch_env(base: Path, side: str, label: str, payload_name: str) -> tuple[dict[str, str], Path, Path]: + root = base / side / label / payload_name + metrics_dir = root / "metrics" + state_dir = root / "state" + metrics_dir.mkdir(parents=True, exist_ok=True) + state_dir.mkdir(parents=True, exist_ok=True) + findings_path = root / "findings.json" + findings_path.write_text("[]", encoding="utf-8") + env = os.environ.copy() + env["CATSTACK_HOOK_METRICS_DIR"] = str(metrics_dir) + env["CATSTACK_HOOK_FINDINGS_FILE"] = str(findings_path) + env.pop("CATSTACK_HOOK_PAYLOAD_DIR", None) + for name in STATE_ENVS: + env[name] = str(state_dir / name.lower()) + return env, metrics_dir, findings_path + + +def _run_command(spec: CommandSpec, payload: bytes, payload_name: str, side: str, scratch: Path, timeout: float) -> dict: + env, metrics_dir, findings_path = _scratch_env(scratch, side, spec.label, payload_name) + timed_out = False + try: + proc = subprocess.run( + spec.command, + shell=True, + input=payload, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=env, + timeout=timeout, + ) + stdout = proc.stdout + stderr = proc.stderr + exit_code = proc.returncode + except subprocess.TimeoutExpired as exc: + timed_out = True + stdout = exc.stdout or b"" + stderr = (exc.stderr or b"") + f"replay_hook_payloads: timed out after {timeout}s\n".encode() + exit_code = 1 + result = outcome.classify(exit_code, stdout, stderr, timed_out) + return { + "command": spec.command, + "exit_code": exit_code, + "outcome": result, + "blocked": result == "blocked", + "context": _context(stdout), + "findings": _rule_ids(metrics_dir, findings_path), + "stdout_b64": base64.b64encode(stdout).decode("ascii"), + "stdout_sha256": hashlib.sha256(stdout).hexdigest(), + "stdout_text": stdout.decode("utf-8", errors="replace"), + "stderr_b64": base64.b64encode(stderr).decode("ascii"), + "stderr_text": stderr.decode("utf-8", errors="replace"), + } + + +def _by_label(specs: list[CommandSpec], name: str) -> dict[str, CommandSpec]: + result: dict[str, CommandSpec] = {} + for spec in specs: + if spec.label in result: + raise SystemExit(f"duplicate {name} label: {spec.label}") + result[spec.label] = spec + return result + + +def _comparison_keys(verdict: dict) -> dict: + return { + "exit_code": verdict["exit_code"], + "blocked": verdict["blocked"], + "context": verdict["context"], + "findings": verdict["findings"], + "stdout_b64": verdict["stdout_b64"], + } + + +def _payload_files(payload_dir: Path) -> list[Path]: + return sorted(path for path in payload_dir.glob("*.json") if path.is_file()) + + +def _print_table(rows: list[dict]) -> None: + print("payload detector verdict fleet_exit dispatcher_exit fleet_outcome dispatcher_outcome") + for row in rows: + verdict = "match" if row["match"] else "mismatch" + print( + f"{row['payload']} {row['detector']} {verdict} " + f"{row['fleet']['exit_code']} {row['dispatcher']['exit_code']} " + f"{row['fleet']['outcome']} {row['dispatcher']['outcome']}" + ) + + +def _parse_args(argv: list[str]) -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--payload-dir", required=True) + parser.add_argument("--fleet", action="append", type=_parse_command_spec, required=True) + parser.add_argument("--dispatcher", action="append", type=_parse_command_spec, required=True) + parser.add_argument("--timeout", type=float, default=30.0) + parser.add_argument("--report-json") + parser.add_argument("--keep-scratch", action="store_true") + return parser.parse_args(argv) + + +def main(argv: list[str] | None = None) -> int: + args = _parse_args(sys.argv[1:] if argv is None else argv) + payload_dir = Path(args.payload_dir) + payloads = _payload_files(payload_dir) + if not payloads: + raise SystemExit(f"no recorded payloads in {payload_dir}") + fleet = _by_label(args.fleet, "fleet") + dispatcher = _by_label(args.dispatcher, "dispatcher") + if set(fleet) != set(dispatcher): + missing_dispatcher = sorted(set(fleet) - set(dispatcher)) + missing_fleet = sorted(set(dispatcher) - set(fleet)) + raise SystemExit( + f"fleet/dispatcher labels differ: missing dispatcher={missing_dispatcher} missing fleet={missing_fleet}" + ) + + scratch = Path(tempfile.mkdtemp(prefix="catstack-hook-replay-")) + rows: list[dict] = [] + try: + for payload_path in payloads: + payload = payload_path.read_bytes() + for label in sorted(fleet): + fleet_verdict = _run_command(fleet[label], payload, payload_path.name, "fleet", scratch, args.timeout) + dispatcher_verdict = _run_command( + dispatcher[label], payload, payload_path.name, "dispatcher", scratch, args.timeout + ) + rows.append( + { + "payload": payload_path.name, + "detector": label, + "match": _comparison_keys(fleet_verdict) == _comparison_keys(dispatcher_verdict), + "fleet": fleet_verdict, + "dispatcher": dispatcher_verdict, + } + ) + finally: + if args.keep_scratch: + print(f"scratch={scratch}", file=sys.stderr) + else: + shutil.rmtree(scratch) + + report = {"payload_dir": str(payload_dir), "match": all(row["match"] for row in rows), "rows": rows} + if args.report_json: + with open(args.report_json, "w", encoding="utf-8") as handle: + json.dump(report, handle, indent=2, sort_keys=True) + handle.write("\n") + _print_table(rows) + return 0 if report["match"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/test/test_replay_hook_payloads.py b/scripts/test/test_replay_hook_payloads.py new file mode 100644 index 00000000..a8925a5e --- /dev/null +++ b/scripts/test/test_replay_hook_payloads.py @@ -0,0 +1,99 @@ +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +import shlex +import subprocess +import sys +import tempfile +import unittest + +SCRIPT = Path(__file__).resolve().parent / "replay_hook_payloads.py" + + +class ReplayHookPayloadsCLI(unittest.TestCase): + def setUp(self) -> None: + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.root = Path(self.tmp.name) + self.payload_dir = self.root / "payloads" + self.payload_dir.mkdir() + self.hook = self.root / "hook.py" + self.hook.write_text( + "import json\n" + "import os\n" + "import sys\n" + "payload = sys.stdin.buffer.read().decode('utf-8')\n" + "mode = sys.argv[1]\n" + "with open(os.environ['CATSTACK_HOOK_FINDINGS_FILE'], 'w', encoding='utf-8') as handle:\n" + " json.dump(['fixture.rule'], handle)\n" + "if mode == 'block':\n" + " print(json.dumps({'decision': 'block', 'reason': payload}))\n" + "elif mode == 'exit2':\n" + " sys.stderr.write(payload + '\\n')\n" + " sys.exit(2)\n" + "else:\n" + " print(json.dumps({'hookSpecificOutput': {'additionalContext': mode + ':' + payload}}))\n", + encoding="utf-8", + ) + + def _payload(self, body: bytes = b'{"hello":"world"}') -> str: + name = f"{hashlib.sha256(body).hexdigest()[:12]}-fixture.json" + (self.payload_dir / name).write_bytes(body) + return name + + def _command(self, mode: str) -> str: + return f"{shlex.quote(sys.executable)} {shlex.quote(str(self.hook))} {shlex.quote(mode)}" + + def _run(self, fleet_mode: str, dispatcher_mode: str) -> subprocess.CompletedProcess[str]: + report = self.root / "report.json" + return subprocess.run( + [ + sys.executable, + str(SCRIPT), + "--payload-dir", + str(self.payload_dir), + "--fleet", + f"fixture={self._command(fleet_mode)}", + "--dispatcher", + f"fixture={self._command(dispatcher_mode)}", + "--report-json", + str(report), + ], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + + def test_matching_commands_emit_match_table_and_json_report(self) -> None: + payload_name = self._payload() + + result = self._run("same", "same") + + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn(f"{payload_name} fixture match 0 0 spoke spoke", result.stdout) + with open(self.root / "report.json", encoding="utf-8") as handle: + report = json.load(handle) + self.assertTrue(report["match"]) + self.assertEqual(report["rows"][0]["detector"], "fixture") + self.assertEqual(report["rows"][0]["fleet"]["findings"], ["fixture.rule"]) + + def test_mismatching_context_returns_nonzero_and_marks_the_row(self) -> None: + self._payload() + + result = self._run("fleet", "dispatcher") + + self.assertEqual(result.returncode, 1, result.stdout + result.stderr) + self.assertIn("fixture mismatch 0 0 spoke spoke", result.stdout) + with open(self.root / "report.json", encoding="utf-8") as handle: + report = json.load(handle) + self.assertFalse(report["match"]) + self.assertNotEqual( + report["rows"][0]["fleet"]["context"], + report["rows"][0]["dispatcher"]["context"], + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_cat_mode.py b/tests/test_cat_mode.py index 7577dbc0..7a822888 100644 --- a/tests/test_cat_mode.py +++ b/tests/test_cat_mode.py @@ -202,7 +202,8 @@ def test_verify_reference_carries_full_original_session_rule(self): class TestPublishingAnalyticsDefaults(unittest.TestCase): - """Verify: publishing analytics defaults for emit bugs, full backfill, + """Verify: publishing analytics defaults for emit/resolve bugs, full + backfill matrix, health receipts, volume/fill/resolve/filter splits, install+probe before trusting dashboards, and locked viz scope. """ @@ -216,12 +217,22 @@ class TestPublishingAnalyticsDefaults(unittest.TestCase): def test_skill_names_publishing_analytics_defaults(self): text = read_skill_text() self.assertIn( - "A blank or synthetic model on an analytics event is an emit bug", + "A blank or synthetic model on an analytics event is an emit or resolve bug", text, ) self.assertIn("never hide it with a chart filter", text) + self.assertIn("never treat a blank-include chart change as the fix", text) self.assertIn( - '"Full backfill" means every emitter of the metric', + "A multi-emitter backfill is done only with a harness×source matrix plus fill table", + text, + ) + self.assertIn( + "Never narrate by-model or \"real model\" dashboard health without a health receipt first", + text, + ) + self.assertIn("filled_model_rate", text) + self.assertIn( + "Split volume vs fill vs resolve vs filter before any L3 or UI insight PATCH when a harness looks missing or a real-model tile reads zero", text, ) self.assertIn( @@ -236,12 +247,29 @@ def test_verify_reference_carries_full_publishing_analytics_rules(self): text = handle.read() self.assertIn("## Publishing analytics", text) self.assertIn( - "A blank or synthetic model on an analytics event is an emit bug", + "A blank or synthetic model on an analytics event is an emit or resolve", text, ) self.assertIn("Do not work around it by filtering", text) + self.assertIn("blank-include", text) + self.assertIn("live resolve path", text) + self.assertIn( + "A multi-emitter backfill is done only with a harness×source matrix", + text, + ) + self.assertIn("fill table", text) + self.assertIn("full backfill", text) + self.assertIn("all harnesses", text) + self.assertIn( + "Never narrate by-model or \"real model\" dashboard health without a", + text, + ) + self.assertIn("filled_model_rate", text) + self.assertIn("volume vs fill vs resolve vs filter", text) + self.assertIn("missing harness X", text) + self.assertIn("real model = 0", text) self.assertIn( - '"Full backfill" means every emitter of the metric', + "Split volume vs fill vs resolve vs filter before any L3 or UI insight", text, ) self.assertIn( @@ -262,6 +290,42 @@ def test_fires_blank_model_chart_filter_workaround(self): self.assertIn("filter out", text.lower()) self.assertIn("blank", text.lower()) + def test_fires_blank_include_chart_workaround(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_blank_include_chart_workaround.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("include blank", text) + self.assertIn("resolve", text) + + def test_fires_dashboard_health_without_receipt(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_dashboard_health_without_receipt.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("health is good", text) + self.assertIn("filled_model_rate", text) + + def test_fires_missing_harness_ui_patch_before_split(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_missing_harness_ui_patch_before_split.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("insight", text) + self.assertIn("volume vs fill vs resolve vs filter", text) + + def test_fires_full_backfill_without_matrix(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_full_backfill_without_matrix.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("codex only", text) + self.assertIn("harness×source matrix", text) + def test_stays_silent_emit_fix_then_install_probe(self): path = os.path.join( self.FIXTURES_DIR, "stays_silent_emit_fix_then_install_probe.md" @@ -271,6 +335,16 @@ def test_stays_silent_emit_fix_then_install_probe(self): self.assertIn("backfill", text.lower()) self.assertIn("probe", text.lower()) + def test_stays_silent_health_receipt_then_diagnose(self): + path = os.path.join( + self.FIXTURES_DIR, "stays_silent_health_receipt_then_diagnose.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("health receipt", text) + self.assertIn("filled_model_rate", text) + self.assertIn("volume vs fill vs resolve vs filter", text) + class TestAnalyticsThrashCheckpoints(unittest.TestCase): """Verify/autonomy: thrash checkpoints for analytics backfills — @@ -302,6 +376,10 @@ def test_skill_names_thrash_checkpoint_defaults(self): self.assertIn("Schema before UI", text) self.assertIn("Claim a chart gap fixed only after naming write vs enrich ownership", text) self.assertIn("Write a thrash checkpoint before attaching new scope", text) + self.assertIn( + "When a same-type miss forces a working-style catch mid analytics thrash, stop feature PRs until that catch is locked", + text, + ) self.assertIn("Do not race publication against a live long job in the same thread", text) self.assertIn( "A plan pivot parks the partial tree before new scope", @@ -317,6 +395,12 @@ def test_verify_reference_carries_analytics_thrash_checkpoints(self): self.assertIn("Schema before UI", text) self.assertIn("Claim a chart gap fixed only after naming write vs enrich ownership", text) self.assertIn("Write a thrash checkpoint before attaching new scope", text) + self.assertIn("intervention-must-automate", text) + self.assertIn( + "When a same-type miss forces a working-style catch mid analytics thrash, stop feature PRs until that catch is locked", + text, + ) + self.assertIn("same-type miss forces a working-style catch", text) self.assertIn( "Million-row or multi-host backfills are owned by a durable runner", text, @@ -324,6 +408,15 @@ def test_verify_reference_carries_analytics_thrash_checkpoints(self): self.assertIn("[[principle-push-not-poll]]", text) self.assertIn("not another check-wait-repeat count", text) + def test_fires_automate_mid_analytics_without_locking_catch(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_automate_mid_analytics_without_locking_catch.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("automate-me", text) + self.assertIn("defer locking the class catch", text) + def test_autonomy_reference_carries_done_gate_publication_and_pivot(self): with open(self.AUTONOMY_REF, encoding="utf-8") as handle: text = handle.read() @@ -390,11 +483,13 @@ def test_every_referenced_skill_still_exists(self): "safe-stack-push", # Command / process tokens in backticks, not skill names. "checkout", + "intervention-must-automate", "invoker-cli", "kill", "ls", "owner-serve", "strace", + "intervention-must-automate", # Absolute-negative words in backticks, not skill names. "only", "never",