diff --git a/corpus/skills/cat-mode/SKILL.md b/corpus/skills/cat-mode/SKILL.md index 4e0ffcc4..03af40d6 100644 --- a/corpus/skills/cat-mode/SKILL.md +++ b/corpus/skills/cat-mode/SKILL.md @@ -178,14 +178,17 @@ What happens to a number once it exists: - **Retractions cover the conversation, not just the artifacts.** - **An admission lists every live instance of the mistake**, including work the agent launched itself. - **A claim about the repo's own history is a query, not a recollection.** -- **A blank or synthetic model on an analytics event is an emit bug.** Fix it at write or backfill — never hide it with a chart filter. [[principle-assert-invariants-not-last-bug]], [[principle-explicit-errors]]. -- **"Full backfill" means every emitter of the metric**, not the first harness that was easy. Same class as Harness-agnostic product defaults above. +- **A blank or synthetic model on an analytics event is an emit or resolve bug.** Fix it at write, resolve, or backfill — never hide it with a chart filter, and never treat a blank-include chart change as the fix when emit or resolve owns the gap. Prove the live resolve path for that harness first. [[principle-assert-invariants-not-last-bug]], [[principle-explicit-errors]]. +- **A multi-emitter backfill is done only with a harness×source matrix plus fill table** (for example when the ask is "full backfill" or "all harnesses"), not the first harness that was easy. Same class as Harness-agnostic product defaults above. +- **Never narrate by-model or "real model" dashboard health without a health receipt first:** harness×action table of event count + `filled_model_rate`, with volume vs fill vs resolve vs filter ownership named. +- **Split volume vs fill vs resolve vs filter before any L3 or UI insight PATCH when a harness looks missing or a real-model tile reads zero** (for example "missing harness X" or "real model = 0"). - **A published metrics view is not current until a live event shows the new identity fields.** Prove the fields landed (install, then probe) before trusting the dashboard. Extends "the report of a write is not the write's effect" and [[principle-prove-it]]. - **Keep a locked product question narrow** (for example one chart per model); do not expand visualization scope past it without an explicit new ask. Follow `narrow-the-scope`. - **Climb analytics tiers in order:** L1 explain → L2 one canary query → L3 backfill/fleet only after the canary proves an identity or cardinality gap. - **Schema before UI.** Required identity properties plus a green canary before the first insight or notebook PATCH; split the forward-stamp PR from the historical-backfill PR. - **Claim a chart gap fixed only after naming write vs enrich ownership** (local vs peer vs fleet). - **Write a thrash checkpoint before attaching new scope:** after two same-type interventions or the first live backfill, record tier and canary status first. +- **Stop analytics feature iteration and lock the class catch before more analytics PRs when automate is mandated mid thrash** (for example the first `intervention-must-automate` flag, an explicit `/automate-me`, or a same-type restatement). Each rule's full text: [references/verify.md](references/verify.md). diff --git a/corpus/skills/cat-mode/references/verify.md b/corpus/skills/cat-mode/references/verify.md index 85ee7b0e..c112a33d 100644 --- a/corpus/skills/cat-mode/references/verify.md +++ b/corpus/skills/cat-mode/references/verify.md @@ -149,18 +149,37 @@ Standing defaults when publishing hook/skill analytics — identity fields, backfills, dashboards, and charts. Prefer fixing emit and coverage over chart workarounds. -- **A blank or synthetic model on an analytics event is an emit bug.** Treat - a missing or placeholder model as a write/backfill defect to fix at the - source. Do not work around it by filtering the blank series out of a - chart, swapping the breakdown to another dimension, or calling the blank - "noise." The model field is an invariant of the event, not a display - preference. [[principle-assert-invariants-not-last-bug]], +- **A blank or synthetic model on an analytics event is an emit or resolve + bug.** Treat a missing or placeholder model as a write, resolve, or + backfill defect to fix at the source. Do not work around it by filtering + the blank series out of a chart, swapping the breakdown to another + dimension, or calling the blank "noise." Do not treat a blank-include + chart change as the fix when emit or resolve owns the gap — prove the + live resolve path for that harness first (same-session peers from local + metrics when the payload omits the model). The model field is an + invariant of the event, not a display preference. + [[principle-assert-invariants-not-last-bug]], [[principle-explicit-errors]]. -- **"Full backfill" means every emitter of the metric.** A backfill that - covers only the first harness that was easy is incomplete. Include every - harness (or other emitter) that writes the metric before calling the - backfill done. Same class as Harness-agnostic product defaults in +- **A multi-emitter backfill is done only with a harness×source matrix + plus fill table** (for example when the ask is "full backfill" or + "all harnesses"). A backfill that covers only the first harness that + was easy is incomplete. Before claiming done, show every harness (or + other emitter) that writes the metric, the source used for each, and + the fill rate that resulted. Scoping to one harness without that matrix + is incomplete. Same class as Harness-agnostic product defaults in SKILL.md. +- **Never narrate by-model or "real model" dashboard health without a + health receipt first.** Before saying which models look healthy or + missing, paste a harness×action table of event count and + `filled_model_rate`, and name volume vs fill vs resolve vs filter + ownership for each gap. A chart anecdote is not that receipt. + [[principle-prove-it]], [[principle-explicit-errors]]. +- **Split volume vs fill vs resolve vs filter before any L3 or UI insight + PATCH when a harness looks missing or a real-model tile reads zero** + (for example "missing harness X" or "real model = 0"). Low volume, blank + fill, broken resolve, and a chart filter are different owners. Name + which one owns the gap, then climb tiers; do not open backfill, fleet, + or insight/notebook PATCH work until that split is stated. - **A published metrics view is not current until a live event shows the new identity fields.** Shipping a chart or notebook update is not proof the new fields land. Prove they landed by installing the emitting path, @@ -195,6 +214,12 @@ defaults above and [[principle-push-not-poll]]; they do not replace them. of a run, write the current tier and canary status before attaching new stack publication or plan scope. A checkpoint is a receipt, not a status ping. +- **Stop analytics feature iteration and lock the class catch before more + analytics PRs when automate is mandated mid thrash** (for example the + first `intervention-must-automate` flag, an explicit `/automate-me`, or + a same-type restatement). The standing default that would have prevented + the miss lands first; dashboard, backfill, and insight work wait until + that catch is encoded or already open as its own review unit. - **Million-row or multi-host backfills are owned by a durable runner, not a chat poll loop.** Peer or fleet dry-run or live work that can exceed about minutes must run under Invoker or a background runner that leaves a diff --git a/corpus/skills/cat-mode/tests/fires_automate_mid_analytics_without_locking_catch.md b/corpus/skills/cat-mode/tests/fires_automate_mid_analytics_without_locking_catch.md new file mode 100644 index 00000000..640eb514 --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_automate_mid_analytics_without_locking_catch.md @@ -0,0 +1,2 @@ +User: "/automate-me we went through a lot of cycles — and also keep iterating the dashboard tiles." +Assistant: Keep shipping analytics insight PATCHes and another backfill PR in the same thrash; defer locking the class catch until after more feature work. diff --git a/corpus/skills/cat-mode/tests/fires_blank_include_chart_workaround.md b/corpus/skills/cat-mode/tests/fires_blank_include_chart_workaround.md new file mode 100644 index 00000000..6a6148b9 --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_blank_include_chart_workaround.md @@ -0,0 +1,2 @@ +User: "Why is cursor w real model = 0 on skill tiles?" +Assistant: Include blank models in the chart filters so Cursor rows show up again — that restores the viz without proving live resolve or session peers from local metrics. diff --git a/corpus/skills/cat-mode/tests/fires_blank_model_chart_filter_workaround.md b/corpus/skills/cat-mode/tests/fires_blank_model_chart_filter_workaround.md index 5a8447b2..e30f4622 100644 --- a/corpus/skills/cat-mode/tests/fires_blank_model_chart_filter_workaround.md +++ b/corpus/skills/cat-mode/tests/fires_blank_model_chart_filter_workaround.md @@ -1,2 +1,2 @@ User: "Why do so many rows show blank model on the hooks chart?" -Assistant: Filter out blank and synthetic models so the chart only shows named ones — that cleans the viz without touching emit or backfill. +Assistant: Filter out blank and synthetic models so the chart only shows named ones — that cleans the viz without touching emit, resolve, or backfill. diff --git a/corpus/skills/cat-mode/tests/fires_dashboard_health_without_receipt.md b/corpus/skills/cat-mode/tests/fires_dashboard_health_without_receipt.md new file mode 100644 index 00000000..7fececd7 --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_dashboard_health_without_receipt.md @@ -0,0 +1,2 @@ +User: "Is the by-model dashboard healthy? Cursor real model looks odd." +Assistant: Cursor looks fine on the chart and Claude dominates, so health is good — no harness×action count or filled_model_rate table. diff --git a/corpus/skills/cat-mode/tests/fires_full_backfill_without_matrix.md b/corpus/skills/cat-mode/tests/fires_full_backfill_without_matrix.md new file mode 100644 index 00000000..4495f3bd --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_full_backfill_without_matrix.md @@ -0,0 +1,2 @@ +User: "Full backfill all harnesses, then we're done." +Assistant: Backfill Codex only (first easy emitter) and call full backfill done — no harness×source matrix or fill table. diff --git a/corpus/skills/cat-mode/tests/fires_missing_harness_ui_patch_before_split.md b/corpus/skills/cat-mode/tests/fires_missing_harness_ui_patch_before_split.md new file mode 100644 index 00000000..786b55e9 --- /dev/null +++ b/corpus/skills/cat-mode/tests/fires_missing_harness_ui_patch_before_split.md @@ -0,0 +1,2 @@ +User: "Why are we missing so many Codex? Also why is Cursor real model = 0?" +Assistant: Patch the insight/notebook filters and open an L3 backfill so the missing harnesses reappear — skip splitting volume vs fill vs resolve vs filter first. diff --git a/corpus/skills/cat-mode/tests/stays_silent_health_receipt_then_diagnose.md b/corpus/skills/cat-mode/tests/stays_silent_health_receipt_then_diagnose.md new file mode 100644 index 00000000..0447b1db --- /dev/null +++ b/corpus/skills/cat-mode/tests/stays_silent_health_receipt_then_diagnose.md @@ -0,0 +1,2 @@ +User: "Why are we missing so many Codex? Also why is Cursor real model = 0?" +Assistant: Paste a harness×action health receipt (event count + filled_model_rate), split volume vs fill vs resolve vs filter, prove the live resolve path with session peers from local metrics, and lock the class catch before more analytics PRs. diff --git a/tests/test_cat_mode.py b/tests/test_cat_mode.py index 7577dbc0..98626800 100644 --- a/tests/test_cat_mode.py +++ b/tests/test_cat_mode.py @@ -202,7 +202,8 @@ def test_verify_reference_carries_full_original_session_rule(self): class TestPublishingAnalyticsDefaults(unittest.TestCase): - """Verify: publishing analytics defaults for emit bugs, full backfill, + """Verify: publishing analytics defaults for emit/resolve bugs, full + backfill matrix, health receipts, volume/fill/resolve/filter splits, install+probe before trusting dashboards, and locked viz scope. """ @@ -216,12 +217,22 @@ class TestPublishingAnalyticsDefaults(unittest.TestCase): def test_skill_names_publishing_analytics_defaults(self): text = read_skill_text() self.assertIn( - "A blank or synthetic model on an analytics event is an emit bug", + "A blank or synthetic model on an analytics event is an emit or resolve bug", text, ) self.assertIn("never hide it with a chart filter", text) + self.assertIn("never treat a blank-include chart change as the fix", text) self.assertIn( - '"Full backfill" means every emitter of the metric', + "A multi-emitter backfill is done only with a harness×source matrix plus fill table", + text, + ) + self.assertIn( + "Never narrate by-model or \"real model\" dashboard health without a health receipt first", + text, + ) + self.assertIn("filled_model_rate", text) + self.assertIn( + "Split volume vs fill vs resolve vs filter before any L3 or UI insight PATCH when a harness looks missing or a real-model tile reads zero", text, ) self.assertIn( @@ -236,12 +247,29 @@ def test_verify_reference_carries_full_publishing_analytics_rules(self): text = handle.read() self.assertIn("## Publishing analytics", text) self.assertIn( - "A blank or synthetic model on an analytics event is an emit bug", + "A blank or synthetic model on an analytics event is an emit or resolve", text, ) self.assertIn("Do not work around it by filtering", text) + self.assertIn("blank-include", text) + self.assertIn("live resolve path", text) + self.assertIn( + "A multi-emitter backfill is done only with a harness×source matrix", + text, + ) + self.assertIn("fill table", text) + self.assertIn("full backfill", text) + self.assertIn("all harnesses", text) + self.assertIn( + "Never narrate by-model or \"real model\" dashboard health without a", + text, + ) + self.assertIn("filled_model_rate", text) + self.assertIn("volume vs fill vs resolve vs filter", text) + self.assertIn("missing harness X", text) + self.assertIn("real model = 0", text) self.assertIn( - '"Full backfill" means every emitter of the metric', + "Split volume vs fill vs resolve vs filter before any L3 or UI insight", text, ) self.assertIn( @@ -262,6 +290,42 @@ def test_fires_blank_model_chart_filter_workaround(self): self.assertIn("filter out", text.lower()) self.assertIn("blank", text.lower()) + def test_fires_blank_include_chart_workaround(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_blank_include_chart_workaround.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("include blank", text) + self.assertIn("resolve", text) + + def test_fires_dashboard_health_without_receipt(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_dashboard_health_without_receipt.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("health is good", text) + self.assertIn("filled_model_rate", text) + + def test_fires_missing_harness_ui_patch_before_split(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_missing_harness_ui_patch_before_split.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("insight", text) + self.assertIn("volume vs fill vs resolve vs filter", text) + + def test_fires_full_backfill_without_matrix(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_full_backfill_without_matrix.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("codex only", text) + self.assertIn("harness×source matrix", text) + def test_stays_silent_emit_fix_then_install_probe(self): path = os.path.join( self.FIXTURES_DIR, "stays_silent_emit_fix_then_install_probe.md" @@ -271,6 +335,16 @@ def test_stays_silent_emit_fix_then_install_probe(self): self.assertIn("backfill", text.lower()) self.assertIn("probe", text.lower()) + def test_stays_silent_health_receipt_then_diagnose(self): + path = os.path.join( + self.FIXTURES_DIR, "stays_silent_health_receipt_then_diagnose.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("health receipt", text) + self.assertIn("filled_model_rate", text) + self.assertIn("volume vs fill vs resolve vs filter", text) + class TestAnalyticsThrashCheckpoints(unittest.TestCase): """Verify/autonomy: thrash checkpoints for analytics backfills — @@ -302,6 +376,10 @@ def test_skill_names_thrash_checkpoint_defaults(self): self.assertIn("Schema before UI", text) self.assertIn("Claim a chart gap fixed only after naming write vs enrich ownership", text) self.assertIn("Write a thrash checkpoint before attaching new scope", text) + self.assertIn( + "Stop analytics feature iteration and lock the class catch before more analytics PRs when automate is mandated mid thrash", + text, + ) self.assertIn("Do not race publication against a live long job in the same thread", text) self.assertIn( "A plan pivot parks the partial tree before new scope", @@ -317,6 +395,12 @@ def test_verify_reference_carries_analytics_thrash_checkpoints(self): self.assertIn("Schema before UI", text) self.assertIn("Claim a chart gap fixed only after naming write vs enrich ownership", text) self.assertIn("Write a thrash checkpoint before attaching new scope", text) + self.assertIn("intervention-must-automate", text) + self.assertIn( + "Stop analytics feature iteration and lock the class catch before more", + text, + ) + self.assertIn("when automate is mandated mid thrash", text) self.assertIn( "Million-row or multi-host backfills are owned by a durable runner", text, @@ -324,6 +408,15 @@ def test_verify_reference_carries_analytics_thrash_checkpoints(self): self.assertIn("[[principle-push-not-poll]]", text) self.assertIn("not another check-wait-repeat count", text) + def test_fires_automate_mid_analytics_without_locking_catch(self): + path = os.path.join( + self.FIXTURES_DIR, "fires_automate_mid_analytics_without_locking_catch.md" + ) + with open(path, encoding="utf-8") as handle: + text = handle.read().lower() + self.assertIn("automate-me", text) + self.assertIn("defer locking the class catch", text) + def test_autonomy_reference_carries_done_gate_publication_and_pivot(self): with open(self.AUTONOMY_REF, encoding="utf-8") as handle: text = handle.read() @@ -390,6 +483,7 @@ def test_every_referenced_skill_still_exists(self): "safe-stack-push", # Command / process tokens in backticks, not skill names. "checkout", + "intervention-must-automate", "invoker-cli", "kill", "ls",