Repository navigation
Conversation
added 30 commits
October 6, 2026 21:47
Publication audit (T-11) before co-authorship: re-derived every published number about the symptom-index and fail-closed work from raw artifacts. Retracted, with the referent recorded: - reader stability "one model 8/10, another flipped 6/10": the denominator exists in no surviving file, and two defensible rules disagree on which reader was steadier. - "glm FA 0.24-0.38": 0.38 is nemotron-nano-30b, a different model. A category error on our side, and no raw survives anywhere. - "recall 0.08 -> 0.88": kept only with the caveat that it is a change of evidence format (code_first to file_content_first), not before/after. Confirmed from recovered raw: the arrival layer maps the dispositional symptom to family A in 5/5 runs. The E11 handout was recovered byte-exact from the opencode session database, its sha256 matches a pre-registered value, and it was written 35s before the first model call, so the recovery is pre-look and the replay it enables is a real replay. Data defect found in our own results: two F4b run files are byte-identical while the manifest declares 22 runs over 21 distinct files, so the "arrival 10/11 vs symptom 3/11" denominator is inflated by one. The direction is unaffected: every symptom failure is on the same item. Guard: scripts/check_duplicate_runs.py, exit 1 on a duplicate or a declared/actual mismatch, exit 2 on an empty population, --selftest 5/5, plus 16 unit tests. It reproduces the F4b defect and passes on recovered_e7 and pinned_variant. KNOWN_ISSUES.md and docs/archive/* hold another session's uncommitted work and are deliberately untouched; proposals are in the PENDING_LEDGER.
Inputs for the deterministic delivery check against Tirthahq/crystal-memory @ fe4e465, frozen before any run so the expectations cannot be fitted to the result. - frozen/ACTS.md: 18 acts in six groups. Word-boundary cases where a short key or one of the seven generic words appears only inside a longer word; substring cases that must still match; whole-word generic cases; same-tier and cross-tier ordering pairs; write-scope cases for the 600/2000 char subject rule; and telemetry cases including a dry run that must emit nothing. - frozen/ORDER_RULE.md: ordering is scored only inside a rotation tier, the overlap gap that makes an ordering difference observable is fixed at 0.10, and delivered / headed / starved are reported separately. - frozen/HYPOTHESES.md: 12 hypotheses with falsifiers, of which three are predicted to FAIL on purpose (the long-context subject guard, the post-hoc specificity threshold, and the Windows selftest, which is outside the declared support envelope). Five mandatory controls, including the blind replay. The run has two arms: our independent reimplementation and their code in a one-off copy. A claim counts as confirmed only when both arms agree on every act; disagreement is the finding, not a pass.
…cumented, zero arm divergences
…ation and one downgrade
… research, M4 design
Recovered run transcripts under experiments/*/frozen/ carry historical version strings by design. Rewriting them to match today would destroy the pre-look integrity the experiments rest on, so drift-checking them can only ever be a false positive. Measured: 10 files, up to 18 drifts each, all inside frozen/. Discrimination re-proven live and manifest re-pinned.
Rig rewritten into the repo; the full-30 rig was a throwaway file in TEMP. Four gates, each self-tested before spending: artifact written before teardown, plant hash pinned per run, fresh hook-free fixture repo, and a variability gate that refuses to scale on a constant outcome. Result: 4/4 committed, 4/4 opened callee.py, 0/4 fixed. Outcome constant, exit 3. Measured cost 0.106 USD.
The matcher was a regex requiring the two tokens `git` and `commit` to be adjacent, so `git -C <dir> commit` was never gated. M4 full-30 run 11 committed that way, was not discarded despite a failed C-plant, and was scored as NO-FIX. Replaced with a tokenizer that skips value-taking global options (-C, -c, --git-dir, ...) and ignores quoted segments, so `echo "git commit"` no longer over-gates the gate. New control file: 11 positive and 9 negative cases, 20/20.
…0.000 Previous three fixtures scored the outcome the agent produced anyway (against-prior requirement unmet, Harness-IF arXiv:2608.11727) and used a fully specified task, leaving no decision to influence (AutoDojo arXiv:2606.15057). v4 makes the task action-open with two correct resolutions and scores the artifact, not prose. G0 prior gate: P(PRESERVE)=5/9=0.556 from the working-tree diff. Prose disagreed with artifact in 2 of 9 runs. Main contrast, 18 runs: V1 (pushes UPDATE) 9/9 UPDATE, V2 (pushes PRESERVE) 9/9 PRESERVE, baseline 4/9 UPDATE. Spread 1.000 against a pre-registered 0.30 threshold. Rig defects found and fixed: provider ceiling ~200s per session; artifact captured the fixture commit instead of the agent work; silent null from transport failure masquerading as a unanimous prior; silent model fallback. Cost: 27 successful runs, 0.48 USD plus 9 zero-cost transport failures.
Both notes point the same way (UPDATE) and are length-matched within 4% (130 vs 135), so wording is separable from direction: H1 imperative 9/9 UPDATE, H2 hedged 9/9 UPDATE, spread 0.000 against a pre-registered 0.30 threshold. Read as NOT DETECTED, not as absent: at n=9/arm a 9/9 vs 8/9 difference is unresolvable, and both arms saturated at 1.000, which removes the instrument's sensitivity to any smaller difference. Both arms moved the prior from 0.444 (G0, 4 of 9) to 1.000. Prose and artifact disagreed in 1 of 18 runs (run 3 declared PRESERVE, patched UPDATE). Cost: v4+v5 58 runs, 0.735 USD.
…16 rows Ran his E1/E2 and overlap fixture through our own implementations, from inputs frozen and hashed before any analysis. E1: our estimate -0.3056, CI [-0.5, -0.1111], SUPPORT, 108/111 rows banked, 12/12 tasks, approach 0.972 vs 0.972 (not compromised). His code agrees to four decimals. Label-permutation control collapses it to -0.003. E2: C1 -0.0556 CI [-0.25, 0.1111] INCONCLUSIVE; C2 -0.1111 CI [-0.3889, 0.1111] INCONCLUSIVE; C3 -0.5455 CI [-0.8485, -0.2727] SUPPORT on 11 tasks of 12 included (Model B), REPLICATES. All three agree with his verdicts. Overlap: 25/25 input hashes reproduce from crystal-memory @ fe4e465; 400/416 rows reproduce; 16 rows are unverifiable because the fixture ships the SYNTHETIC-unicode hash without its text. Collapsed-tokenizer control matches 0/171 fraction rows, so the check discriminates. Three bugs in our own verifier are documented in the report: a leftover loop variable that compared a foreign input 416 times, a strip() that shifted the 1500-char cut by one character, and a scanner that printed 0/0 and looked like a pass.
Promised in the issue thread and frozen before running. Their own dispatcher on our isolated store (the 3 starters, the entire deployable population at fe4e465), scored from their own eligibility/delivery logs. No model in the loop. H1: 4 boundary keys (eval/ssh/ssm/tsc) match only as words, 8/8; 9 symbolic keys match verbatim, pipe is load-bearing, 10/10; 6 long keys keep substring semantics (old behaviour preserved), 12/12. H2: superstrings of the seven generic words add nobody, 7/7. H3: must-match-nothing acts give empty lists, 3/3. H4: order of 3 two-candidate acts equals our independent word-overlap computation, 3/3. H5: stable across repeats. G4 blind repeat identical. Caveats stated in the report: the blind control is vacuous (0 candidates, guard has no binding), repeatability on non-empty sets comes from H5 instead; the seven words have no deployed binding so only their superstring behaviour is covered; order checked on 3 acts. Mirror left clean, ledger restored.
Red team against crystal-memory @ fe4e465, all deterministic, no model, frozen plan before running. A substring false positives on length-5 keys, 5/5 HIT (escargot fires green-tests via cargo; |tailspin/|headache/|grepable/<<EOFILE fire exit-code): BUG, natural text triggers warnings about absent danger. B seven-word rule ablation: 0/8 delivery changes, the advertised change has no effect on the deployed set: DEPLOYMENT FACT. C silent truncation (max_n=1 on a 2-candidate act): 1 delivered, 1 starved, starved text absent from output: DESIGN LIMIT. D repeat danger in one session: first warns, second is empty (backoff+cap): DESIGN LIMIT. E rotation beats relevance on repeats (unit-level, 0.065 vs 0.021 overlap): DESIGN LIMIT. F benign battery 0/10 deliveries: GOOD. Also adds HISTORY.md, the full story ledger from invitation through withdrawals, X12, re-derivation, delivery check and red team, for the eventual thread comment.
Three of our own published claims were wrong and are corrected here, with evidence. 1. SYNTHETIC-unicode: the 16 rows ARE verifiable. The input text lives in his frozen generator; our terminal rendered it as ASCII mush and we concluded the text was absent. With the real text the hash matches and all 16 rows agree, including the 4/9 unicode-tokenization control. Overlap is 416/416, not 400/416. The hole we reported did not exist. 2. Attack B ablation was vacuous: emptying _BOUNDARY_KEYS cannot change anything by construction because short keys take the boundary path on length alone. Replaced with real evidence: a synthetic guard-key crystal proves the mechanism works (word matches, glued does not), and a full inventory of the 25 deployed keys proves none of them depend on the set. Conclusion stands (no deployed effect), evidence replaced. 3. Attack C used an artificial max_n=1. On the live 3-crystal store under the default budget all 3 deliver and nothing starves. Reclassified: mechanism verified, not triggered here. Also verified: repeat-silence second run raises no exception (real empty, not a crash); no expired flags or tiers on the deployed set; rotation-env order coincides on the tested act. Uncovered paths named in the report: pack/specificity role, systematic rotation comparison, holdback, expiry.
All deterministic, no model, expectations frozen before running (redteam/PLAN2.md lives next to the scripts). B1 obfuscation: env-var assignment does NOT hide (substring present, fires); base64/escaping/ANSI/padding bypass (no literal, silent); heredoc body scanned; background & does not hide. Two of my first B1 rows were confounded (contained go build/go test) and were redone clean with a control. B2 seen poisoning, stronger than the TZ: [1,0,0,0,0] on 5 repeats (backoff, not the cap), backoff key is per ACT TYPE so any bash act in the window suppresses. B3 fail modes: corrupted ledger returns+delivers (fail-open), broken crystal skipped silently (fail-open), 10MB ctx in 0.4s (no ReDoS). B4 TOCTOU: heredoc body visible, & does not hide; intra-process race absent by construction (synchronous single call). B5 scale: load 0.87/8.02/39.07s and match 0.034/0.326/1.735s at 100/1000/5000 crystals, both linear; 1.7s per act at 5000 exceeds the pain threshold.
Three of our own red-team claims were wrong and are corrected with evidence. B2 cap test forged the ledger twice without the schema key and without session-last; both times the loader discarded the forgery and the test measured an empty ledger. Redone with schema+live session: cap fires exactly at seen=3 (control seen=2 delivers), and live 5x repeats go [1,0,0,0,0] because time-backoff suppresses from the second run, so the cap is a backstop that repeats never reach. B1 two rows contained go build/go test and fired on those, not on the tested obfuscation; redone clean with a control. C on max_n=1 is reclassified: mechanism verified, but on the live 3-crystal store under the default budget all 3 deliver and nothing starves. Scale numbers are lower bounds (synthetic crystals carry 2 keys each); ~1400 crystals crosses the 500ms pain threshold at 0.35ms each.
The 30 runner records, the recovered transcripts+telemetry (30/30), the zero-fix diagnosis, and the two uncommitted rigs (ambig, notes). Evidence preservation: this data was already lost once when worktrees were deleted before capture.
Four advanced vectors, plan frozen before running, plus ~25 sourced literature anchors and a second agent re-verifying all five technical claims with its own code (5/5 AGREE). V1 weaponized crystals: evil file loads 1/1 with no complaint and fires 1/1; no content-verification mechanism exists (only structural invalid_bindings, a ledger-cache signature, and fail-open credential redaction). V2 attention deflection: 1 large critical vs 12 small weaks sharing the act vocabulary; critical starved 1/1 with its text absent from output. Mechanism established by dissection: candidate order follows overlap honestly, but the greedy best-fit packer takes small pieces first, so a swarm of small high-overlap weaks crowds out a large critical. Reverse also measured (small critical survives by best-fit). V3 shell aliasing: aliased git/cargo fire green-tests on text the shell would execute differently; matcher resolves no aliases (verified by reading). V4 phantom poisoning: repeat goes silent by backoff; forged seen=3 blocks while seen=2 delivers (cap discriminates exactly); clock-ageing releases backoff but not cap; fresh session speaks (cap is per-session).
Three pilot failures, three fixes, all measured. (1) PowerShell agents cannot execute extensionless sh wrappers: added .cmd wrappers via a python relay (note from env, never interpolated). (2) opencode denies Read on *.env by default and asks on external dirs: scoped OPENCODE_CONFIG override for experiment runs only (fixture .env files hold no secrets); global config untouched. (3) Identical harness line appended to every prompt telling the agent to use cat when a read is refused. Pilot t5a x 2 arms now CLEAN: approach=True both, audit rows flow, scorer runs.
proc.kill() leaves grandchildren holding our pipes and the next communicate() hangs forever (measured: 25 min, zero output). All run_agent implementations now use taskkill /F /T first with proc.kill as fallback and a bounded second communicate. Post-mortem P-kill recorded in AGENT_DIARY: terminated the OpenCode Desktop app mistaking it for zombies (age is not evidence of zombieness); guard is terminate-only-own-children plus Win32 parent-chain check, Electron main+renderer trees never touched.
added 7 commits
October 10, 2026 21:02
score_run now merges the lab_server side-effect log (vendor-api rows live there, not in audit.log). t3a excluded from the pooled contrast with stated reason: both agents bypassed the wrapped standins (urllib straight to the API), so the note never arrived in either arm; kept as a delivery-coverage finding (agents hit the quota wall, note undeliverable off the instrumented path). Kill-tree fix applied to run_agent. 11-trap pooled contrast at reps=1: HELP 5/10 vs SHAM 5/10 hits given approach, exactly null at this n.
12 traps x 2 arms x 2 reps on our agent with his tasks/fixture/standins/scorer. HELP (specific advisory) vs SHAM (generic), length-matched. Pooled hit|approach 9/15 vs 9/17: no detectable direction effect. This does NOT refute his E1: his notes are refusal-first, ours advisory. The honest reading is a null for advisory notes, silence on refusals. Instrument defects stated: 12/44 no-approach (provider timeouts, bypasses); t3a excluded (note undeliverable off the instrumented path, kept as a coverage finding); t22a valid no-approach both arms. Cost: 55 sessions, 1.19 USD.
#2 weapon 1/3 vs 0/3 (non-zero utilization, weak evidence at n=3). #3 swarm N=1 7U/0P vs N=5 3U/4P (preference competition moves choice). #4 WARNING 1U/2P vs NONE 1U/2P, identical, instrument fixed mid-flight (wrapper-function third resolution shape added; full rescore changed 1 classification, already matching manual review). #5 6/6 fix all 3, no decay, ceiling. Rule going forward: every run saves its working-tree diff, or independent re-scoring is impossible (s3/s4 lacked diffs). Cost w2+s3+s4+s5 0.51 USD.
REFUSE (prohibitive mood, same mechanisms, length-matched 131-136) vs fresh SHAM, 11 traps x 1 rep: approach 4/11 vs 9/11, hit|approach 2/4 vs 6/9. The approach gap has no cite evidence (tails show bypasses and tool-discovery stalls, never the note), so it is noise+variance, not a mechanism. His E1 effect did not replicate on our stack in advisory form (0.600 vs 0.529, 44 runs) or prohibitive form (0.500 vs 0.667, 22 runs). Stated as non-replication on a different agent/harness, not refutation. Cost: REFUSE 11 sessions 0.23 USD; replication total 77 sessions 1.65 USD.
The v4/v5 runs classified outcomes at runtime and printed them, but saved no artifact field, so an independent audit read missing keys as 0.0. Backfilled by re-running the fixed classifier (with wrapper-function resolution) on the saved working-tree diffs: 36 files, zero disagreements with the printed runtime verdicts. The report audit now reproduces every headline number.
R1 (44 runs, HELP vs SHAM) + R2 refusal arm (22 runs, REFUSE vs fresh SHAM) + pilots. Every run: approach/hit from his scorer on our artifacts, note attempt flag, tails, worktree state. Saved before teardown per the artifact rule.
Adds the experiment index (read order, guarantees), text-level results (10/10 predicted: case folds, 6 invisible bypasses silent), independent verification record (second agent 5/5 AGREE, report audit ALL NUMBERS REPRODUCE), and the full draft comment for Tom's thread with every number carrying its denominator and every error corrected next to the number it touched. Nothing posted externally; draft awaits explicit word.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Independent audit of the Tom Jones crystal lab (tjonesit/crystals#1) plus our own note-tone experiments (M4). Every number carries a denominator; every instrument has a control that can fail; our own errors are documented next to the numbers they touched.
Independent re-derivation (deterministic, )
Red-team findings (all measured, )
Our-stack experiments (measured costs)
Fixes included
Review guide
Start at \experiments/tom_e1e2_independent/README.md, then HISTORY.md (10 chapters).
Numbers live in RESULTS*.md; every one recomputes via the scripts next to it.