Hooks: load the writing grader when installed; stop seven false blocks and fix the session audit by EdbertChan · Pull Request #1403 · EdbertChan/catstack

EdbertChan · 2026-10-14T06:22:39Z

Summary

Before Claude sends a reply, small checkers read it and may block it. A separate audit tool reviews a whole session afterward.

The problem: one checker could not load its grader and blocked every document reply unread. Six more blocked plain prose, a clock command, scratch files, and known words.

The cause: the installed copy sits outside the repo, word lists in error text read as capability claims, and quotes, variable assignments, and older turns were ignored.

The fix: find the grader from the install record, log a failed load instead of blocking, honor quotes and assignments, remember every user word, and skip files outside a repo.

The session audit now flags a checker that is not checking anything.

Review Claim

Eight checkers and the session audit stop blocking or miscounting the cases listed below, and each change ships with a fixture that fails on the old code and a positive fixture that still passes.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

No checker gains a new block. Every change narrows when a checker fires or widens what the audit reports, and each narrowing carries a negative fixture that failed before plus a positive fixture that still passes. A grader that cannot load now logs an unchecked row and lets the turn through, which is the written-down fail direction for that check.

Assumptions: Safety Invariant unconfirmed, headless run.

Slice Rationale

All eight fixes share one evidence set (one session's transcript and hook metrics) and one gate run, so they ship as one PR in the engine unit. Each fix is its own commit, so any one can be reverted alone:

technical-writing-guard: grader resolved via the snapshot's source record, then the installed skill dirs, then the old walk; import failure goes to the unchecked channel. Fixture: the Stop entrypoint run from a copy of the hook under a fake ~/.cache/catstack-hooks-snapshot/ prints "could not load" on the old code and grades the bad and good documents on the new code.
hook-health: a hook with three or more rows since the last prompt, all unchecked, gets its own notice line; those rows stop counting as "blocked on purpose". The SDK runner writes unchecked rule ids to the findings file.
hedge-runs-prove-it: capability values come only from labelled, identifier-shaped lists in this turn's errors. Fixture: "The step times only live in GitLab job logs; the parser shows ci_step rows" after a Read that mentions an error blocks on the old detector with (logs, parser) and is silent now; [gpt-5.5, gpt-5.5-pro] after a real error still blocks.
wait-needs-wakeup: date is never a repeat; wait words inside a quoted span or in a reply naming the hook are not a wait. Fixture: a third date +%H:%M\ %Z exits 2 on the old code and 0 now; a third identical curl still exits 2.
no-comments: files outside a git work tree are not judged. Fixture: a Write of an HTML page with comment lines under a temp dir with no .git returns one finding on the old code and none now; the same page under a dir with .git still fires.
diu-stop: words from every human turn form an allow-list; a hit whose content words the user already wrote is dropped. Fixture: the user writes "MRs" seven turns before the reply "The MR database has 3 rows."; the judge's hit on "MR database" blocks on the old code and is dropped now, while a term the user did not write still blocks.
reflect token_audit: an Artifact publish of an edited file ends the no-verify edit streak (fixture: Edit, Edit, publish, Edit, Edit, publish reads 4 on the old code and 2 now); each cache-creation spike row carries gap_seconds_since_previous_assistant and preceding_user_chars; a new hook-unchecked-blocks flag counts Stop-hook feedback rows that say the check could not run (fixture: three such rows read yes/3, two read no/2). A mid-turn queued human send with no later user row is pinned by a fixture; that part is a regression test, not a behavior change.
prove-it-ship-gate: a leading shell assignment is skipped like sudo. Fixture: K=~/.claude/skills/invoker-plan-to-invoker; ls $K returns ['invoker-plan-to-invoker'] on the old code and [] now; INVOKER_HOME=/tmp/x invoker-cli doctor keeps ['invoker-cli doctor'].

Non-goals

No new hook. No change to how install.sh builds the hook snapshot. The generated block in docs/skill-triggers.md and the ruff F811 findings in engine/hooks/unverified-tag-ledger/tests/test_hooks.py are not touched; neither file is in this diff. The Codex audit loop gets only the artifact rule. The streak semantics for a Bash verify after N edits are unchanged.

Measured

Command: python3 measure_hh.py <hook-health dir> where the script calls detect.notice([row]*3, "claude") on the runs.jsonl row shape {"hook":"technical-writing-guard","outcome":"blocked","exit_code":2,"rule_ids":["technical-writing-guard.unchecked"],"stderr_tail":"The document check could not load. Leave the reply unchecked.\n","harness":"claude","event":"Stop"}

base: None
head: hook-health: technical-writing-guard reported unchecked on all 3 runs since the last prompt: The document check could not load. Leave the reply unchecked.; it is not checking anything.

Test Plan

All hook suites run under source scripts/test/hermetic_env.sh.

Revert Plan

Safe to revert? Yes
Revert command: git revert <first-sha>^..<last-sha> for the eight commits, or revert any single commit alone
Post-revert steps: rerun ./install.sh so the hook snapshot picks up the reverted detectors
Data migration? No

🤖 Generated with Claude Code

…and log an import failure as unchecked The installed copy of this hook lives in ~/.cache/catstack-hooks-snapshot, so walking three directories up from detect.py landed in ~/.cache and the grade_prose import failed on every run. That failure came back as a blocking Finding, so the hook blocked replies without ever grading one: 201 of 276 technical-writing-guard rows in runs.jsonl since 2026-10-13T10:24Z carry rule technical-writing-guard.unchecked (115 of 167 when the reflect pass measured it). The grader is now resolved through source_repo(), then the installed skill under ~/.claude, ~/.cursor, ~/.codex, then the old walk. An ImportError goes to the _catstack_unchecked_findings channel, so the runner logs action=unchecked and does not block. Tests: test_runs_from_a_snapshot_outside_the_repo copies the hook, _sdk, _flags and hooks.toml into a fake snapshot under a temp HOME and runs the Stop entrypoint; it failed with "could not load" before the change and grades the bad and good fixtures after it. test_grader_import_failure_is_unchecked_not_blocked pins the fail-open path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…last prompt hook-health only counted crashed, timed_out and caught_error rows as failures, and counted every blocked row as "blocked on purpose". So 80 identical technical-writing-guard rows with rule_ids ["technical-writing-guard.unchecked"] were reported as the hook doing its job, while the hook had not graded a single reply. A hook with three or more rows since the last prompt, all of them unchecked (every rule id ends in .unchecked, or action == "unchecked"), now gets its own notice line naming the stderr text, once per hook until that text changes, and those rows no longer count as blocked on purpose. The SDK runner also writes unchecked rule ids into the findings file, so a fail-open check still shows its unchecked rule in runs.jsonl. Tests: test_hook_unchecked_on_every_run_is_reported and test_unchecked_rows_logged_as_action_are_reported returned None before the change; test_single_unchecked_run_stays_quiet, test_hook_that_also_checked_something_stays_quiet and test_unchecked_notice_is_not_repeated_once_seen pin the silent cases. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…entifier-shaped lists in this turn's errors The capability check harvested every comma-joined run from any tool result that merely contained an error word (a Read of a SKILL.md qualified), across the whole transcript, and then matched those words near "only" in the reply. One session hit seven false blocks on word lists like (asks, box, kept, seen), (label, we), (pasted, visual), (420, spec), (logs, parser, plus). This adapts unmerged commit 6093b8db from branch reflect/hook-false-positives-20261008 (cherry-picked clean onto main): a list counts only when the error text introduces it ("Known models:", "valid values:", "one of"), only from this turn's tool results, and only for identifier-shaped values (a letter plus an inner -_./:@+). Plain words between commas are prose. Tests: the two session shapes (test_silent_on_only_live_in_logs_after_a_skill_read_that_mentions_error, test_silent_on_read_only_question_after_an_error_with_label_we) blocked with "(logs, parser)" and "(arrow, label)" on the old detector and are silent now; test_still_blocks_known_models_after_a_real_error keeps the [gpt-5.5, gpt-5.5-pro] positive green. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…nside quotes or about the hook are not a wait The Stop message tells the agent to run `date +%H:%M\ %Z` for an ETA, but the PreToolUse repeat rule blocked the third identical shell command with no allowlist, so the agent quoted a clock 4.5 hours old. And is_wait_reply only skipped a WAIT_RE match whose previous character was a quote, so a review of this hook that wrote 'the reply said "it would report back"' was itself blocked as a wait. A command matching ^\s*(TZ=\S+\s+)?date(\s|$) is exempt from the repeat rule. is_wait_reply skips any match inside a double-quoted, backticked or single-quoted span (the containment check hedge-runs-prove-it uses) and returns False when the reply names wait-needs-wakeup. Tests: test_third_date_command_is_not_a_repeat exited 2 before and 0 now; test_describing_a_blocked_reply_is_not_waiting returned the block message before and None now; test_third_identical_curl_is_still_a_repeat and test_unquoted_report_back_is_still_waiting keep the positives green. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The hook blocked a Write of a session scratchpad page under /private/tmp (11 HTML comment lines) and the agent routed around it with another tool. The block's own reason, that the commit message and git blame carry the story, does not hold for a file outside any repository. detect() now walks up from the file's nearest existing parent looking for a .git entry, with no subprocess, and skips the file when none is found. A path whose nearest existing parent is the filesystem root is still judged, so the existing fixtures at /repo and /r keep firing. Tests: test_scratchpad_html_outside_a_repo_is_not_checked returned one finding before the change and none after; test_html_inside_a_repo_still_fires and test_file_at_an_unknown_location_is_still_checked pin the two judged cases. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

plain_words handed the judge only the last five human turns, although the prompt says a word the user used first does not count. So it blocked "MR database" and "model MR (!22502)" after the user had written "MRs" at rows 551 and 1318 of the transcript, and "force-pushed with lease" after the user had used those words. job() now builds a vocabulary of lower-cased, lightly stemmed content words from every human turn (same row filtering as before, no limit), sends it to the judge as an allow-list, and check_reply drops a hit whose every content word the user has already written. Tests: test_plain_words_skips_a_term_the_user_used_six_turns_ago blocked before the change (the prompt's USER block held only "next question 1..5") and is silent now; test_a_term_the_user_never_used_still_blocks keeps the positive green; test_limit_agreement.py still passes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…s carry the idle gap, and a new hook-unchecked-blocks flag Four measurement gaps from one session audit. (a) A human message typed mid-turn is an enqueue queue-operation row that is later removed with reason absorbed_mid_turn and never gets its own user row. The current counting path already keeps it (the real transcript's row 1537 is in direct_human_claude_rows, n_user_messages=23 includes it), so this commit pins that shape with test_absorbed_mid_turn_send_is_counted_once rather than changing the count; the test passes on the unmodified code. (b) no-verify-edit-streak flagged 4 on an HTML page whose every edit batch was followed by an Artifact publish of that same file. Publishing the page is that file's run step, like `python3 foo.py` after editing foo.py, so an Artifact call naming an edited file now ends the streak in both the Claude and Codex loops and is counted as "artifact publishes=N". On the real session the flag goes from 4/yes to 2/no with artifact publishes=5. test_artifact_publish_of_the_edited_page_ends_the_streak read 4 before. (c) cache-creation-spikes now carries a details row per spike with the gap in seconds since the previous assistant turn and the size of the user row before it, so an idle-gap rebuild (8,087 s and 50,015 s here) reads apart from a big payload without re-reading the transcript. (d) New flag hook-unchecked-blocks counts Stop-hook feedback rows whose text says the check could not run, with the hook name read from the runner's `--timeout N <hook>/` string; yes at three or more. The real session reads 18 (technical-writing-guard=18). Documented in references/cost-audit.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

surface_keys() took the first word of each unquoted segment as the program, so `K=~/.claude/skills/invoker-plan-to-invoker; ls $K` yielded basename "invoker-plan-to-invoker", which matches CLI_PROGRAM_RE. The hook then blocked five replies in one session demanding a rerun of "implementation-stack" and "invoker-plan-to-invoker", neither of which is a command. Leading words shaped like an assignment (^[A-Za-z_][A-Za-z0-9_]*=) are now skipped the same way sudo/env are, and the program after them is still read, so `INVOKER_HOME=/tmp/x invoker-cli doctor` keeps its key. Tests: test_surface_keys_skip_shell_variable_assignments returned ['invoker-plan-to-invoker'] before the change and [] after; the existing surface_keys assertions stay green. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

EdbertChan and others added 8 commits October 13, 2026 22:37

EdbertChan added the admin-bypass label Oct 19, 2026

Navigation Menu

Repository navigation

Hooks: load the writing grader when installed; stop seven false blocks and fix the session audit - #1403

Hooks: load the writing grader when installed; stop seven false blocks and fix the session audit#1403
EdbertChan wants to merge 8 commits into
mainfrom
reflect/hook-paths-and-false-positives-20261013

EdbertChan commented Oct 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

EdbertChan commented Oct 14, 2026

Summary

Review Claim

Review Lane

Review Unit

Safety Invariant

Slice Rationale

Non-goals

Measured

Test Plan

Revert Plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant