Skip to content

fix(server): a run that failed to start no longer wedges a nearly full thread - #21011

Closed
stebo85 wants to merge 4 commits into
pingdotgg:mainfrom
stebo85:fix/handoff-budget-missed-run-retry
Closed

stebo85 wants to merge 4 commits into
pingdotgg:mainfrom
stebo85:fix/handoff-budget-missed-run-retry

Conversation

@stebo85

@stebo85 stebo85 commented Oct 17, 2026 •

Copy link
Copy Markdown

A run that fails before reaching its provider can make every later request fail on a nearly full native thread: replaying the missed request exceeds the context-handoff allowance even though the native conversation can still accept the new request.

The start service retries delivery with required handoffs only when optional missed-run recovery cannot fit in the same native thread. It records the missed items as omitted after the provider accepts the new turn. Required handoffs retain their budget failures. Budget calculations, contracts and provider adapters are unchanged.

Continues @stebo85's work in #21011, merged onto current main while preserving the author's commits. The rebuild replaces a zero-or-one array and its mapped persistence with one optional handoff, and explicitly requires reuse of the same native thread. This stays limited to failed-start retries.

Verification:

  • Retained one integration scenario in the existing ProviderSwitch suite: 69 cases before → 69 after; all 68 main cases preserved. It proves the retry and following request complete in the original native thread, recovery is omitted once, and a required handoff still fails. With main's start service, the retry assertion fails; with the fix, all 69 pass.
  • Five relevant server suites: 124 tests pass. Scoped lint/format, server typecheck and whitespace checks pass.
  • One full-diff review round by Claude Opus 5.5 and GPT-6-Sol: both found no blocking code defects. Their publication gates are addressed by the committed-head live replay and this updated description; new-head CI is tracked after pushing.
  • Live headless Linux app with real Codex: first turn completed, a missing image caused an actual start failure before a native provider turn existed, main rejected the retry with the budget error, and the rebuilt server accepted it on the same native thread. Further failed starts and retries exercised the committed same-native guard; the following request completed without another handoff. Persisted runs and omission records were inspected. Live verification summary.

The near-full usage report was simulated at 200,000/258,400 tokens in isolated test state; the native conversation itself was not organically filled. The failures, provider starts, replies and omission bookkeeping ran through the real server and Codex adapter. Required handoff rejection is covered by the integration harness, not a live provider switch.

Before — main refuses the retry:

Before: real Codex thread rejects a retry with the simulated near-full usage report

After — guarded retry completes in the original conversation:

After: real Codex accepts the guarded retry after another failed start

Not verified: an organically full Codex conversation, macOS, Windows, iOS, Android, or Electron desktop. Human maintainer review remains required.

Implemented through T3 Code's Codex harness with GPT-6.1-Sol; tests by GPT-6-Astra; independent reviews by GPT-6-Sol and Claude Opus 5.5 through T3 Code.

Ubuntu and others added 2 commits October 17, 2026 21:23
…window

A Codex thread at 366k/475k tokens whose previous run failed to start
refuses every later turn with ContextHandoffBudgetError. This test fails
until the fix lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l thread

When a run fails before the provider sees it, the next turn on the same
native thread replays the missed items as a retry handoff. The handoff
budget reserves a quarter of the window on top of measured usage, so a
thread past about 75% full had no budget for even the recovery pointer.
Every later turn failed with ContextHandoffBudgetError and added another
missed run, until the user compacted.

The missed items are optional context for a native thread that already
holds the conversation. When they cannot fit, deliver the required
handoffs alone and record the retry handoff as omitted once the provider
accepts the turn, so it is not offered again. Required handoffs still
fail with ContextHandoffBudgetError when they do not fit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Oct 17, 2026
@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Path: .coderabbit.config.ts
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8a090399-f365-4780-b4d7-7ceb34180a48
📥 Commits

Reviewing files that changed from the base of the PR and between 13aa882 and 91ae67f.

📒 Files selected for processing: 2
  • apps/server/src/orchestration-v2/ProviderTurnStartService.ts
  • apps/server/src/orchestration-v2/testkit/ProviderSwitch.integration.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

Provider-turn start now retries context handoff delivery after a budget error when retry handoffs and a native thread ID are available. Integration coverage checks that the missed request is omitted from the settled delta handoff and that later requests complete without another handoff.

Changes: Context handoff retry

Layer / File(s) Summary
Retry handoff delivery
apps/server/src/orchestration-v2/ProviderTurnStartService.ts, apps/server/src/orchestration-v2/testkit/ProviderSwitch.integration.test.ts
The service reuses delivery input and retries with effective handoffs when retry handoffs and a native thread ID are present. It records retry handoffs as inline with their history message IDs omitted. The integration test covers the retry and a subsequent request.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Suggested reviewers: juliusmarminge

Merge Risk: ⚪ Minimal · up to 91ae6

The nearly full thread can retry without repeatedly offering the failed run’s handoff. No actionable merge-blocking issue remains after normal checks.

Security architecture risk: 🔵 Low · up to 91ae6

The recovery path preserves required context transfers and existing execution permissions. Failed-request context is deliberately omitted only after a later turn is accepted. No new privilege or cross-thread access path was identified, but fresh-thread recovery and interrupted settlement retain some uncertainty.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The inspected change affects context presented to the selected provider session and delivery coverage stored for that application thread and native identity. Retry selection does not introduce a new cross-thread history source. This bounds the demonstrated exposure without establishing repository-wide tenant isolation.

Trust Boundaries and Controls

  • observed — The provider prompt combines the selected user message with delivered historical context. The existing history wrapper explicitly labels historical material as context rather than a new request or higher-priority instructions. The fallback removes optional history instead of granting it additional authority, and retains the current-attempt check before provider start.

Resilience and Maintainability Implications

  • observed — Resume failure or uncertain prior injection creates a required full-thread recovery handoff. The reduced retry path retains that handoff and its budget failure behavior. The first budget-error attempt fails before injection or pending-record persistence, avoiding partially applied delivery before the reduced attempt.

Hardening Proposals

  • proposed — Make the intended omission policy explicit for newly created native threads as well as resumed ones. The current guard accepts either; a focused recovery scenario could establish whether permanent retry omission is intended when no prior native conversation exists. This is a clarification proposal, not a demonstrated security defect.
Pre-merge checks: Passed 4
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly and concisely describes the fix for failed starts that wedge nearly full native threads.
Description check Passed The description explains the problem, change, scope, verification results, limitations, and testing evidence. It does not use the template headings and does not provide an explicit maintainer approval…
Full details: Description check

Explanation

The description explains the problem, change, scope, verification results, limitations, and testing evidence. It does not use the template headings and does not provide an explicit maintainer approval comment, but the required information is mostly present.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@juliusmarminge juliusmarminge self-assigned this Oct 19, 2026
@juliusmarminge

Copy link
Copy Markdown
Member

Taking over this one. I will keep this focused on failed-start retries in nearly full native threads, preserve required handoff failures, and verify the rebuilt change before pushing here.

Copy link
Copy Markdown
Member

Closing in favour of #24703, which makes the same change. Thanks for the work here!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants