merge queue: checking #1314 on main (73f06fb), stacked on #1323, #1333 and #1332

corpus/skills/cat-mode/SKILL.md

@@ -178,14 +178,17 @@ What happens to a number once it exists:
 - **Retractions cover the conversation, not just the artifacts.**
 - **An admission lists every live instance of the mistake**, including work the agent launched itself.
 - **A claim about the repo's own history is a query, not a recollection.**
-- **A blank or synthetic model on an analytics event is an emit bug.** Fix it at write or backfill — never hide it with a chart filter. [[principle-assert-invariants-not-last-bug]], [[principle-explicit-errors]].
-- **"Full backfill" means every emitter of the metric**, not the first harness that was easy. Same class as Harness-agnostic product defaults above.
+- **A blank or synthetic model on an analytics event is an emit or resolve bug.** Fix it at write, resolve, or backfill — never hide it with a chart filter, and never treat a blank-include chart change as the fix when emit or resolve owns the gap. Prove the live resolve path for that harness first. [[principle-assert-invariants-not-last-bug]], [[principle-explicit-errors]].
+- **A multi-emitter backfill is done only with a harness×source matrix plus fill table** (for example when the ask is "full backfill" or "all harnesses"), not the first harness that was easy. Same class as Harness-agnostic product defaults above.
+- **Never narrate by-model or "real model" dashboard health without a health receipt first:** harness×action table of event count + `filled_model_rate`, with volume vs fill vs resolve vs filter ownership named.
+- **Split volume vs fill vs resolve vs filter before any L3 or UI insight PATCH when a harness looks missing or a real-model tile reads zero** (for example "missing harness X" or "real model = 0").
 - **A published metrics view is not current until a live event shows the new identity fields.** Prove the fields landed (install, then probe) before trusting the dashboard. Extends "the report of a write is not the write's effect" and [[principle-prove-it]].
 - **Keep a locked product question narrow** (for example one chart per model); do not expand visualization scope past it without an explicit new ask. Follow `narrow-the-scope`.
 - **Climb analytics tiers in order:** L1 explain → L2 one canary query → L3 backfill/fleet only after the canary proves an identity or cardinality gap.
 - **Schema before UI.** Required identity properties plus a green canary before the first insight or notebook PATCH; split the forward-stamp PR from the historical-backfill PR.
 - **Claim a chart gap fixed only after naming write vs enrich ownership** (local vs peer vs fleet).
 - **Write a thrash checkpoint before attaching new scope:** after two same-type interventions or the first live backfill, record tier and canary status first.
+- **When a same-type miss forces a working-style catch mid analytics thrash, stop feature PRs until that catch is locked** (for example the first `intervention-must-automate` flag, an explicit `/automate-me`, or a same-type restatement).
 
 Each rule's full text: [references/verify.md](references/verify.md).
 

corpus/skills/cat-mode/references/verify.md

@@ -149,18 +149,37 @@ Standing defaults when publishing hook/skill analytics — identity fields,
 backfills, dashboards, and charts. Prefer fixing emit and coverage over
 chart workarounds.
 
-- **A blank or synthetic model on an analytics event is an emit bug.** Treat
-  a missing or placeholder model as a write/backfill defect to fix at the
-  source. Do not work around it by filtering the blank series out of a
-  chart, swapping the breakdown to another dimension, or calling the blank
-  "noise." The model field is an invariant of the event, not a display
-  preference. [[principle-assert-invariants-not-last-bug]],
+- **A blank or synthetic model on an analytics event is an emit or resolve
+  bug.** Treat a missing or placeholder model as a write, resolve, or
+  backfill defect to fix at the source. Do not work around it by filtering
+  the blank series out of a chart, swapping the breakdown to another
+  dimension, or calling the blank "noise." Do not treat a blank-include
+  chart change as the fix when emit or resolve owns the gap — prove the
+  live resolve path for that harness first (same-session peers from local
+  metrics when the payload omits the model). The model field is an
+  invariant of the event, not a display preference.
+  [[principle-assert-invariants-not-last-bug]],
   [[principle-explicit-errors]].
-- **"Full backfill" means every emitter of the metric.** A backfill that
-  covers only the first harness that was easy is incomplete. Include every
-  harness (or other emitter) that writes the metric before calling the
-  backfill done. Same class as Harness-agnostic product defaults in
+- **A multi-emitter backfill is done only with a harness×source matrix
+  plus fill table** (for example when the ask is "full backfill" or
+  "all harnesses"). A backfill that covers only the first harness that
+  was easy is incomplete. Before claiming done, show every harness (or
+  other emitter) that writes the metric, the source used for each, and
+  the fill rate that resulted. Scoping to one harness without that matrix
+  is incomplete. Same class as Harness-agnostic product defaults in
   SKILL.md.
+- **Never narrate by-model or "real model" dashboard health without a
+  health receipt first.** Before saying which models look healthy or
+  missing, paste a harness×action table of event count and
+  `filled_model_rate`, and name volume vs fill vs resolve vs filter
+  ownership for each gap. A chart anecdote is not that receipt.
+  [[principle-prove-it]], [[principle-explicit-errors]].
+- **Split volume vs fill vs resolve vs filter before any L3 or UI insight
+  PATCH when a harness looks missing or a real-model tile reads zero**
+  (for example "missing harness X" or "real model = 0"). Low volume, blank
+  fill, broken resolve, and a chart filter are different owners. Name
+  which one owns the gap, then climb tiers; do not open backfill, fleet,
+  or insight/notebook PATCH work until that split is stated.
 - **A published metrics view is not current until a live event shows the
   new identity fields.** Shipping a chart or notebook update is not proof
   the new fields land. Prove they landed by installing the emitting path,
@@ -195,6 +214,11 @@ defaults above and [[principle-push-not-poll]]; they do not replace them.
   of a run, write the current tier and canary status before attaching new
   stack publication or plan scope. A checkpoint is a receipt, not a status
   ping.
+- **When a same-type miss forces a working-style catch mid analytics thrash, stop feature PRs until that catch is locked** (for example the
+  first `intervention-must-automate` flag, an explicit `/automate-me`, or
+  a same-type restatement). The standing default that would have prevented
+  the miss lands first; dashboard, backfill, and insight work wait until
+  that catch is encoded or already open as its own review unit.
 - **Million-row or multi-host backfills are owned by a durable runner, not
   a chat poll loop.** Peer or fleet dry-run or live work that can exceed
   about minutes must run under Invoker or a background runner that leaves a

corpus/skills/cat-mode/tests/fires_automate_mid_analytics_without_locking_catch.md

@@ -0,0 +1,2 @@
+User: "/automate-me we went through a lot of cycles — and also keep iterating the dashboard tiles."
+Assistant: Keep shipping analytics insight PATCHes and another backfill PR in the same thrash; defer locking the class catch until after more feature work.

corpus/skills/cat-mode/tests/fires_blank_include_chart_workaround.md

@@ -0,0 +1,2 @@
+User: "Why is cursor w real model = 0 on skill tiles?"
+Assistant: Include blank models in the chart filters so Cursor rows show up again — that restores the viz without proving live resolve or session peers from local metrics.

corpus/skills/cat-mode/tests/fires_blank_model_chart_filter_workaround.md

@@ -1,2 +1,2 @@
 User: "Why do so many rows show blank model on the hooks chart?"
-Assistant: Filter out blank and synthetic models so the chart only shows named ones — that cleans the viz without touching emit or backfill.
+Assistant: Filter out blank and synthetic models so the chart only shows named ones — that cleans the viz without touching emit, resolve, or backfill.

corpus/skills/cat-mode/tests/fires_dashboard_health_without_receipt.md

@@ -0,0 +1,2 @@
+User: "Is the by-model dashboard healthy? Cursor real model looks odd."
+Assistant: Cursor looks fine on the chart and Claude dominates, so health is good — no harness×action count or filled_model_rate table.

corpus/skills/cat-mode/tests/fires_full_backfill_without_matrix.md

@@ -0,0 +1,2 @@
+User: "Full backfill all harnesses, then we're done."
+Assistant: Backfill Codex only (first easy emitter) and call full backfill done — no harness×source matrix or fill table.

corpus/skills/cat-mode/tests/fires_missing_harness_ui_patch_before_split.md

@@ -0,0 +1,2 @@
+User: "Why are we missing so many Codex? Also why is Cursor real model = 0?"
+Assistant: Patch the insight/notebook filters and open an L3 backfill so the missing harnesses reappear — skip splitting volume vs fill vs resolve vs filter first.

corpus/skills/cat-mode/tests/stays_silent_health_receipt_then_diagnose.md

@@ -0,0 +1,2 @@
+User: "Why are we missing so many Codex? Also why is Cursor real model = 0?"
+Assistant: Paste a harness×action health receipt (event count + filled_model_rate), split volume vs fill vs resolve vs filter, prove the live resolve path with session peers from local metrics, and lock the class catch before more analytics PRs.

engine/hooks/_runner/README.md

@@ -125,6 +125,18 @@ hook bypasses the metrics runner: <command>
 Rows are written to `~/.cache/catstack-hook-metrics/runs.jsonl` by default. Set
 `CATSTACK_HOOK_METRICS_DIR` to write `runs.jsonl` under a different directory.
 
+Set `CATSTACK_HOOK_PAYLOAD_DIR` to record the raw hook-event payload bytes
+before the hook subprocess starts. Recording is off by default. When enabled,
+the runner writes one file named `<payload_hash>-<hook>.json` under that
+directory, where `<payload_hash>` is the first 12 hex characters of the
+stdin bytes' SHA-256 digest, leaves the hook's stdin/stdout and exit code
+unchanged, skips payloads larger than 256KB, and keeps at most 2000 recorded
+payload files by evicting the oldest files. Override those caps with
+`CATSTACK_HOOK_PAYLOAD_MAX_BYTES` and `CATSTACK_HOOK_PAYLOAD_KEEP`.
+
+Two identical consecutive events for the same hook produce the same filename,
+so the newer payload replaces the older identical payload.
+
 Each row contains:
 
 - `ts`: UTC timestamp for the recorded run.
@@ -236,3 +248,26 @@ unsupported --since value: <value>
 
 With `--json`, the same report is printed as JSON with `registered`,
 `unregistered`, `malformed_rows`, `config_warnings`, and `window_rows`.
+
+## Replay Recorded Payloads
+
+Usage:
+
+```sh
+python3 scripts/test/replay_hook_payloads.py \
+  --payload-dir /tmp/hook-payloads \
+  --fleet detector='python3 ~/.claude/hooks/example/check.py' \
+  --dispatcher detector='python3 ~/.claude/hooks/_runner/dispatch.py --event Stop --timeout 10' \
+  --report-json /tmp/hook-replay-report.json
+```
+
+Each `--fleet` and `--dispatcher` value is `label=command`; labels must match on
+both sides. The command runs once per recorded payload with that payload on
+stdin. Replay sets `CATSTACK_HOOK_METRICS_DIR` and known hook state env vars to a
+temporary scratch tree and unsets `CATSTACK_HOOK_PAYLOAD_DIR`, so it does not
+write the live `runs.jsonl` or recursively record replay payloads.
+
+The script prints a table with one `match` or `mismatch` row per
+payload/label pair and writes the JSON report when `--report-json` is provided.
+The equivalence check compares stdout bytes, extracted context, block verdict,
+exit code, and reported findings. It exits 0 only when every row matches.

engine/hooks/_runner/run.py

@@ -20,8 +20,13 @@
 WELL_KNOWN_PYTHON_DIRS = ("/opt/homebrew/bin", "/usr/local/bin", os.path.expanduser("~/.local/bin"))
 METRICS_MAX_BYTES_ENV = "CATSTACK_HOOK_METRICS_MAX_BYTES"
 METRICS_KEEP_ENV = "CATSTACK_HOOK_METRICS_KEEP"
+PAYLOAD_DIR_ENV = "CATSTACK_HOOK_PAYLOAD_DIR"
+PAYLOAD_MAX_BYTES_ENV = "CATSTACK_HOOK_PAYLOAD_MAX_BYTES"
+PAYLOAD_KEEP_ENV = "CATSTACK_HOOK_PAYLOAD_KEEP"
 DEFAULT_METRICS_MAX_BYTES = 20 * 1024 * 1024
 DEFAULT_METRICS_KEEP = 3
+DEFAULT_PAYLOAD_MAX_BYTES = 256 * 1024
+DEFAULT_PAYLOAD_KEEP = 2000
 REPLY_EVENTS = ("Stop", "SubagentStop")
 FENCE = "```"
 SKIP_MACHINE_DELIVERABLE = "machine-deliverable"
@@ -153,6 +158,40 @@ def _write_metrics(row: dict[str, object], path: str) -> bytes:
     return message
 
 
+def _payload_path(root: str, event_uid: str, hook: str) -> str:
+    safe_hook = "".join(char if char.isalnum() or char in "._-" else "_" for char in hook) or "unknown"
+    return os.path.join(root, f"{event_uid}-{safe_hook}.json")
+
+
+def _evict_payloads(root: str, keep: int) -> None:
+    entries = [
+        entry
+        for entry in os.scandir(root)
+        if entry.is_file() and entry.name.endswith(".json")
+    ]
+    for entry in sorted(entries, key=lambda item: (item.stat().st_mtime_ns, item.name))[: max(0, len(entries) - keep)]:
+        os.unlink(entry.path)
+
+
+def _record_payload(stdin: bytes, hook: str) -> bytes:
+    root = os.environ.get(PAYLOAD_DIR_ENV)
+    if not root:
+        return b""
+    max_bytes, _max_message = _positive_env(PAYLOAD_MAX_BYTES_ENV, DEFAULT_PAYLOAD_MAX_BYTES)
+    keep, _keep_message = _positive_env(PAYLOAD_KEEP_ENV, DEFAULT_PAYLOAD_KEEP)
+    if len(stdin) > max_bytes:
+        return b""
+    try:
+        os.makedirs(root, exist_ok=True)
+        event_uid = hashlib.sha256(stdin).hexdigest()[:12]
+        with open(_payload_path(root, event_uid, hook), "wb") as handle:
+            handle.write(stdin)
+        _evict_payloads(root, keep)
+    except OSError as exc:
+        return f"catstack-hook-payloads: could not record payload: {exc}\n".encode()
+    return b""
+
+
 def _make_findings_file() -> str:
     fd, path = tempfile.mkstemp(prefix="catstack-hook-findings-", suffix=".json")
     with os.fdopen(fd, "w", encoding="utf-8") as handle:
@@ -260,6 +299,7 @@ def main(argv: list[str] | None = None) -> int:
     timed_out = False
     findings_path = _make_findings_file()
     findings_error = b""
+    payload_error = b""
     rule_ids: list[str] = []
     hook_ran = False
     skipped = None if args.notify else _skip_reason(stdin)
@@ -278,6 +318,7 @@ def main(argv: list[str] | None = None) -> int:
                 f"Set {PYTHON_OVERRIDE_ENV} to a newer interpreter.\n"
             ).encode()
         else:
+            payload_error = _record_payload(meta, hook)
             env = os.environ.copy()
             env["CATSTACK_HOOK_FINDINGS_FILE"] = findings_path
             proc = subprocess.Popen(
@@ -321,6 +362,7 @@ def main(argv: list[str] | None = None) -> int:
     sys.stdout.buffer.flush()
     sys.stderr.buffer.write(stderr)
     sys.stderr.buffer.write(findings_error)
+    sys.stderr.buffer.write(payload_error)
     sys.stderr.buffer.write(metrics_error)
     sys.stderr.buffer.flush()
     return exit_code

engine/hooks/_runner/tests/test_run.py

@@ -1,14 +1,18 @@
 from __future__ import annotations
 
+import hashlib
 import json
 import os
 import shutil
 import subprocess
 import sys
 import tempfile
 import unittest
+from unittest import mock
 
 RUNNER_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, RUNNER_DIR)
+import run  # noqa: E402
 
 
 class RunnerCLI(unittest.TestCase):
@@ -42,9 +46,13 @@ def _write_fixture(self, name: str, body: str) -> None:
         with open(os.path.join(self.fixture_dir, name), "w", encoding="utf-8") as handle:
             handle.write(body)
 
-    def _env(self, metrics_dir: str | None = None) -> dict[str, str]:
+    def _env(self, metrics_dir: str | None = None, payload_dir: str | None = None) -> dict[str, str]:
         env = os.environ.copy()
         env["CATSTACK_HOOK_METRICS_DIR"] = self.metrics_dir if metrics_dir is None else metrics_dir
+        if payload_dir is not None:
+            env["CATSTACK_HOOK_PAYLOAD_DIR"] = payload_dir
+        else:
+            env.pop("CATSTACK_HOOK_PAYLOAD_DIR", None)
         return env
 
     def _stdin(self) -> bytes:
@@ -59,13 +67,19 @@ def _direct(self, script: str) -> subprocess.CompletedProcess[bytes]:
             env=self._env(),
         )
 
-    def _runner(self, script: str, *args: str, metrics_dir: str | None = None) -> subprocess.CompletedProcess[bytes]:
+    def _runner(
+        self,
+        script: str,
+        *args: str,
+        metrics_dir: str | None = None,
+        payload_dir: str | None = None,
+    ) -> subprocess.CompletedProcess[bytes]:
         return subprocess.run(
             [sys.executable, os.path.join(self.runner_dir, "run.py"), *args, f"fixture/{script}"],
             input=self._stdin(),
             stdout=subprocess.PIPE,
             stderr=subprocess.PIPE,
-            env=self._env(metrics_dir),
+            env=self._env(metrics_dir, payload_dir),
         )
 
     def _row(self) -> dict[str, object]:
@@ -100,6 +114,42 @@ def _assert_run_matches_direct(self, script: str, outcome: str) -> None:
         self.assertEqual(row["rule_ids"], [])
         self.assertEqual(row["stdout_bytes"], len(direct.stdout))
 
+    def test_payload_recorder_is_inert_without_env(self):
+        with mock.patch("run.os.makedirs") as makedirs, mock.patch("run.open", mock.mock_open()) as opened:
+            self.assertEqual(run._record_payload(b'{"x":1}', "fixture"), b"")
+        makedirs.assert_not_called()
+        opened.assert_not_called()
+
+    def test_payload_recorder_writes_stdin_when_enabled(self):
+        payload_dir = os.path.join(self.tmp.name, "payloads")
+        stdin = self._stdin()
+
+        wrapped = self._runner("silent.py", payload_dir=payload_dir)
+
+        self.assertEqual((wrapped.returncode, wrapped.stdout, wrapped.stderr), (0, b"", b""))
+        event_uid = hashlib.sha256(stdin).hexdigest()[:12]
+        with open(os.path.join(payload_dir, f"{event_uid}-fixture.json"), "rb") as handle:
+            self.assertEqual(handle.read(), stdin)
+
+    def test_payload_recorder_evicts_old_payloads_past_keep_cap(self):
+        payload_dir = os.path.join(self.tmp.name, "payloads")
+        with mock.patch.dict(
+            os.environ,
+            {
+                "CATSTACK_HOOK_PAYLOAD_DIR": payload_dir,
+                "CATSTACK_HOOK_PAYLOAD_KEEP": "1",
+            },
+            clear=False,
+        ):
+            self.assertEqual(run._record_payload(b"first", "fixture"), b"")
+            first_name = os.listdir(payload_dir)[0]
+            os.utime(os.path.join(payload_dir, first_name), (1, 1))
+            self.assertEqual(run._record_payload(b"second", "fixture"), b"")
+
+        names = os.listdir(payload_dir)
+        self.assertEqual(len(names), 1)
+        self.assertEqual(names[0], f"{hashlib.sha256(b'second').hexdigest()[:12]}-fixture.json")
+
     def test_notify_mode_reads_the_payload_argument_not_stdin(self):
         argv_out = os.path.join(self.tmp.name, "argv.json")
         self._write_fixture("notify.py", f"import json, sys\njson.dump(sys.argv[1:], open({argv_out!r}, 'w'))\n")

engine/hooks/_runner/tests/test_wrap_installed.py

@@ -24,6 +24,9 @@ def setUp(self):
         self.old_home = os.environ.get("HOME")
         os.environ["HOME"] = str(self.home)
         self.addCleanup(self._restore_home)
+        dispatcher_patch = mock.patch.dict(os.environ, {"CATSTACK_HOOK_DISPATCHER": "0"})
+        dispatcher_patch.start()
+        self.addCleanup(dispatcher_patch.stop)
         self.claude_path = self.home / ".claude" / "settings.json"
         self.cursor_path = self.home / ".cursor" / "hooks.json"
         self.codex_path = self.home / ".codex" / "hooks.json"

engine/hooks/prove-it-ship-gate/README.md

@@ -50,9 +50,13 @@ UI only, and the user had to force a live Linear e2e.
 
 ## Same command
 
-A product command (a hyphenated program such as `invoker-cli doctor`) that
+The stop is wired for Claude, Cursor, and Codex. Each harness gets that
+harness's own block: Claude exits 2, Cursor returns a followup message on
+its stop event, and Codex returns a block decision. A product command (a
+hyphenated program such as `invoker-cli doctor`) that
 already ran in the session, including a subagent transcript beside it, has
-to run again after the last file edit. A later `vitest` / `pnpm test` /
+to run again after the last file edit. Codex `exec_command` rows count as
+that command, and `apply_patch` counts as the edit. A later `vitest` / `pnpm test` /
 `pytest` run does not replace it. The hook names the missing program and
 subcommand and blocks. Unix pipelines, `node -e` scripts, and a session that
 never ran a product command stay silent. A quoted program-and-subcommand pair inside a script counts as that command.