Repository navigation
AgentDojo E3: the if-statement control arm (one registered run) - #59
Merged
Merged
Conversation
Arm D inserts arm C's words only when the frozen regex fires on a new tool result; no engine. 726 episodes on E2b's suite, model, seed and injections, judged against E2b's recorded arms. E7/E8: every A - D interval includes 0 (the gating alone explains the gain); E9: D ~ C on ASR, D > C on benign utility (+0.134); E10 pass. bench/agentdojo/e3/, BENCHMARKS.md §14.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One registered run of the control §14 listed under "Not tested": arm D, an if-statement that puts arm C's words into the system prompt only on a model call where a tool result since the previous assistant turn matches the same frozen regex; no engine, no state.
e2b.py+ one branch,bench/agentdojo/e3/experiments/e3.py); injections recomputed and checked byte-identical to E2b's before any API call.bench/agentdojo/e3/PREREGISTRATION.md. Smoke gate passed (D 3/5 vs B's recorded 3/5), then 726 episodes once: 0 errors, 0 retries, 0.95 USD by usage.tables/episodes.jsonluntouched; the E2b tables stay byte-identical).Arm D: utility 0.7113, utility under attack 0.4706, ASR 0.2417. Unregistered context row: D sent B's exact prompts on all 97 benign tasks and 9 of 97 outcomes still differed (run-to-run variation at temperature 0).
What it changes: §14 (new E3 paragraph; the if-statement leaves "Not tested") and the root README bullet, which said the engine's contribution is when the words arrive: an if-statement gets the same when on this suite; the engine's own addition here is the replayable strand per stance.
Recompute:
cd bench/agentdojo/e3 && experiments/rerun_e3.sh <python with numpy==1.26.4>(twice, byte-identical; bars sha256173e350a…, resultsdcf9accd…).Limits: one model, one suite version, one attack template the regex matches by design (987/987 injections), one run per arm; D ran on Linux ~25 h after A–C on macOS; adaptive attacks untested.
Draft: not for merge until the maintainer reruns
rerun_e3.sh.