Skip to content

feat(session): add opt-in automatic crash recovery - #3399

Open
adalundhe wants to merge 2 commits into
fix/session-recovery-manual-chatfrom
fix/session-recovery-automatic
Open

adalundhe wants to merge 2 commits into
fix/session-recovery-manual-chatfrom
fix/session-recovery-automatic

Conversation

@adalundhe

@adalundhe adalundhe commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add controller-wide sessions.automaticRecovery, disabled by default.

Opt-in scans admit recovery for confirmed crashed READY Sessions, including idle and waiting Sessions. Already admitted recoveries continue when the flag is disabled. Chat shows automatic progress without manual recovery choices.

All current workflow callers receive the policy explicitly, including the newly merged waiting-expiration test that caused the earlier combined PR's CI type-check failure.

Stack

Split requested in the review discussion.

Part 6 of 6. 35 files, +656/-165 against the immediate parent.

Part Scope Review
1 add durable recovery primitives PR
2 implement manual crashed-runtime recovery PR
3 expose manual crashed-session revert PR
4 report recovery and preserve discarded message references PR
5 offer manual recovery for crashed conversations PR
6 add opt-in automatic crash recovery PR

Parts 2–6 form a GitHub stack on main, so CI runs on each part. A stack cannot include a fork branch, so part 2 targets main and also shows part 1's commit until #3255 merges. Parts 3–6 target their immediate parents for a focused diff. The stack is rebased onto main after #3409, which removed the public Session suspend and resume controls.

Testing

On main after #3409, the full database, Session, gRPC and MCP race suites pass against PostgreSQL, except TestSandboxTemplateCatalog, which needs envtest binaries. Full Go lint reports no issues, and E2E compiles. Node 24.13: typecheck, lint and all 696 unit tests pass. Buf rate-limited remote generation, so pinned local generators produced this part's output; they reproduce official output byte-for-byte.

Live cluster E2E runs in this PR's CI.

Upstream CI runs on this PR as part of the stack on main.

🤖 Generated with Claude Code

Signed-off-by: Ada Lundhe <adalundhe@lundhe.audio>
@adalundhe
adalundhe added this pull request to stack #3421 October 28, 2026 15:17
@adalundhe
adalundhe force-pushed the fix/session-recovery-automatic branch from 2eafd11 to 270ea5c Compare October 28, 2026 15:18
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Oct 28, 2026
@blacksmith-sh

This comment has been minimized.

TestSessionWorkerLossRecovery lists tasks as soon as the second turn's model
call reaches the gate, and it expected that turn to be WORKING. ADK runtimes
record WORKING while their model call is already in flight. In CI, the kagent
runtime's UpdateTask committed after its model call reached the gate, so the
listing read SUBMITTED and the kagent and byo-adk cases failed. codex and claude
record WORKING first and passed.

A turn is durably accepted once its task exists and has not finished, and
recovery fails SUBMITTED and WORKING tasks alike.

Verified with go vet on the e2e package; the suite runs in CI.

Signed-off-by: Ada Lundhe <adalundhe@lundhe.audio>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Oct 28, 2026
FROM session_record i
WHERE ($1::text = '' OR i.id > $1::uuid)
AND i.state = 'RUNTIME_STATE_FAILED' AND i.operation = 'RUNTIME_OPERATION_RESUME'
AND (($3 AND i.state = 'RUNTIME_STATE_READY' AND i.operation = 'RUNTIME_OPERATION_NONE')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100 sessions per minute means slow detection at scale

<div>{automatic
? "Automatic recovery will restore the last saved runtime state. Messages the recovered agent did not retain are marked as failed."
: "Revert to the last runtime snapshot, or start a new conversation from a saved snapshot. Recovery does not replay messages it did not retain."}</div>
{!automatic && (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

failed auto recovery leaves users no revert or fork

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants