Hook fleet (3) scorecard dashboard

[EdbertChan]

Summary

Small checker scripts run each time an agent acts. Each run is logged. This adds a table that shows what those runs cost, per kind of moment.

The problem: a new way of starting the checkers is coming. Nobody could see whether it is cheaper or safer than the old way.

The cause: the log had the numbers, but no view grouped them into old way and new way.

The fix: one new view sorts each logged run into before or after. It then shows how many were started, how long they took, and how often they failed.

It prints text or data only. It does not build a web page or save a file.

Review Claim

A new read-only view groups logged checker runs into old way and new way, and prints cost and failure numbers for each kind of moment.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Hook stdout and exit codes stay the same. The new code runs only when someone asks for --scorecard; it reads runs.jsonl and writes nothing. No hook, runner, or dispatch file is in the diff.

Assumptions: this text comes from the workflow plan and was not confirmed by a person in this headless run.

Slice Rationale

This is one of five claims split out of the Hook fleet dispatcher branch. It only reads the ledger, so it can be reviewed without the dispatcher changes that write the dispatch_path field.

Non-goals

• No --html flag, no summary.json, and no dashboard.html. The workflow claim named those; this diff does not contain them. It ships the --scorecard text and --json table only.
• No server.
• No read of the events ledger. Only runs.jsonl is read.
• No change to any hook, to the runner, or to how hooks are dispatched.
• No change to the other four Hook fleet claims.
• No change to the existing --events, --runs, --judge, --skills, or --harness-gap modes.

Measured

Command: python3.12 engine/hooks/_runner/report.py --scorecard --days 14

• base: report.py: error: unrecognized arguments: --scorecard --days 14 (exit 2, at origin/main 0adc86f)
• head: before PreToolUse 5465 5465 1.0 765.7 765.7 1.3 0.0 0.00 (exit 0, at f74048a, first rows below)

phase event events detector_runs procs/event cpu_ms/event wall_ms/event timeout% crash% spoke-rate
before PostToolBatch 427 427 1.0 339.4 339.4 0.5 0.0 0.00
before PostToolUse 1696 1696 1.0 409.1 409.1 0.1 0.0 0.00
before PostToolUseFailure 12 12 1.0 193.2 193.2 0.0 0.0 0.00
before PreToolUse 5465 5465 1.0 765.7 765.7 1.3 0.0 0.00
before Stop 769 769 1.0 1284.2 1284.2 0.3 0.0 0.00
before UserPromptSubmit 481 481 1.0 3474.8 3474.8 10.4 58.2 0.06

The head run read the live ledger on this machine and printed 12 rows, all in the before phase and none in after:

$ python3.12 engine/hooks/_runner/report.py --scorecard --days 14 | awk 'NR>1{print $1}' | sort | uniq -c
  12 before
$ grep -c '"dispatch_path"' ~/.cache/catstack-hook-metrics/runs.jsonl
0

The after phase is exercised by the fixture tests in the Test Plan, not by this live run.

Test Plan

Test Plan

• python3.12 -m unittest engine.hooks._runner.tests.test_report — Ran 21 tests in 1.428s / OK
• python3.12 -m unittest engine.hooks._runner.tests.test_report -k scorecard -v — test_scorecard_compares_before_and_after_dispatch_path_rows ... ok, test_scorecard_prints_dashboard_columns ... ok, Ran 2 tests in 0.116s / OK
• python3.12 engine/skills/make-pr/scripts/preflight.py --base origin/main --commit-only --dry-run — unit engine-runtime: 2 file(s) / ok preflight passed
• python3.12 scripts/ci/check_hook_test_coverage.py engine/hooks/_runner — check_hook_test_coverage: OK (1 hook(s) checked)
• ruff check --select E9,F engine/hooks/_runner/report.py engine/hooks/_runner/tests/test_report.py — All checks passed!
• Fixture vs live: the two scorecard tests use fixture rows. The Measured command read the live ledger, which printed before rows only.
• Python version: the same test file run with the system python3 (3.9.6) printed ModuleNotFoundError: No module named 'tomllib' and FAILED (failures=20). The runs above use python3.12.

Revert Plan

Revert Plan

• Safe to revert? Yes
• Revert command: git revert <merge sha of this PR>
• Post-revert steps: None. The --scorecard and --days flags stop being accepted.
• Data migration? No. Nothing is written.

🤖 Generated with Claude Code

---

Note

Low Risk
Read-only CLI reporting on existing metrics logs; no hook execution or ledger writes, isolated to --scorecard.

Overview
Adds a --scorecard mode to report.py that reads runs.jsonl and prints a before/after comparison of hook detector cost and reliability, keyed by event type.

Runs without a dispatch_path land in before; runs with dispatch_path (e.g. sdk, subprocess_fallback) land in after. Rows are grouped by phase, event, session, and event_uid, then aggregated into per-(phase, event) metrics: event count, detector runs, processes per event (with post-dispatch counting one main proc plus subprocess fallbacks), CPU and wall time per event, and timeout/crash/spoke rates. Output is a text table or --json.

Also adds --days as an optional positive float window (overrides --since when set) for scorecard and shared time parsing. format_decimal formats one-decimal columns in the table. Unit tests cover before/after math and printed headers.

^{Reviewed by Cursor Bugbot for commit f74048a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 2437a82c-e2e3-4708-ac77-d4af674eb33e)

[mergify]

Queued — the merge queue status continues in this comment ↓.

[EdbertChan]

@Mergifyio queue

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-10-11 19:22 UTC · Rule: admin-bypass · triggered by @EdbertChan with the @mergifyio queue command
• ✅ Checks passed · on draft merge queue: checking #1324 + #1331 + #1334 + #1335 together on main (0d8f2e0) #1338
• ✅ Merged — 2026-10-11 20:23 UTC · at 44e40c9516c19f675f7c3d3ad6d7df4da3209fd2 · squash

This pull request spent 1 hour 53 seconds in the queue, including 59 minutes 58 seconds running CI.

Required conditions to merge

• check-success = lint
• check-success = test
• check-success = validate