[adalundhe]

Summary

Add the storage and Substrate operations needed by manual Session recovery.

• Admit recovery and settle interrupted tasks in one transaction, respecting Actor identity, lifecycle generation and checkpoint fences.
• Reject a crash observed before another lifecycle operation began, such as a concurrent revert that restored the same Actor.
• Retire an idle suspension that the crash interrupted, so its retained claim cannot block recovery.
• Preserve completed results and waits retained by the runtime snapshot.
• Add the Substrate revert call and select its latest completed durable snapshot without changing checkpoint selection.

Stack

Split requested in the review discussion.

Part 1 of 6. 6 files, +544/-45 against the immediate parent.

Part	Scope	Review
1	add durable recovery primitives	PR
2	implement manual crashed-runtime recovery	PR
3	expose manual crashed-session revert	PR
4	report recovery and preserve discarded message references	PR
5	offer manual recovery for crashed conversations	PR
6	add opt-in automatic crash recovery	PR

Parts 2–6 form a GitHub stack on main, so CI runs on each part. A stack cannot include a fork branch, so part 2 targets main and also shows part 1's commit until #3255 merges. Parts 3–6 target their immediate parents for a focused diff. The stack is rebased onto main after #3409, which removed the public Session suspend and resume controls.

Testing

On main after #3409, the full database race suite passes against PostgreSQL, and full Go lint reports no issues. Tests cover settlement, retained history, admission fences, stale crash observations, interrupted idle suspensions, replay idempotency and rollback. Reverting either review fix makes its new tests fail.

Parts 2–6 form a GitHub stack on main, so upstream CI runs on each of them.

Upstream CI runs on this part, which targets main.

[jjamroga]

I have some general feedback on the solution here. An actor could crash for any number of reasons, but graceful recovery is reasonable.

We should introduce a feature (flagged, off by default) which allows an Agent to automatically recover

• This is primarily for delivering a good experience for machine clients (other agents, microservices, etc)
• Not sure whether this should be system wide or on an agent by agent basis (or both)
• Clients will likely retry when experiencing unavailability, and that's wasted work if the Agent can't recover itself
• This is feature flagged because recovery can be complex, and we want some preliminary feedback before making this default behavior

When disabled (default), the user should be given an option on how they want to recover in chat

• There are a few ways to recover: Revert? Fork from last good state?
• If an agent is supposed to automatically recover, they will obviously not be given an option

After rolling back to last good state, users should be made aware of messages that failed to send

• This does not mean we need to revert our A2A task state, just like a text message we should be showing "failed to send"

[adalundhe]

Addressed the recovery feedback in 1c17482 and 72aaed6, and rebased onto main with the snapshot/lifecycle conflicts resolved.

• sessions.automaticRecovery is a controller-wide opt-in, disabled by default. Already admitted recoveries still converge if it is disabled later.
• Manual mode offers Revert and the existing READY-checkpoint fork in chat. Automatic mode shows progress without manual choices.
• Persisted task/message identity pairs mark discarded user inputs as “Failed to send” without rewinding completed A2A tasks or their history.

The full PostgreSQL store/migration race suites, Go lint, protobuf generation/compatibility, E2E compilation, and focused UI tests/typecheck pass locally. The final broader backend run and current-head CI are still pending. The PR description now reflects this scope and distinguishes the local controlled-runtime checks from live CI E2Es.

[EItanya]

Taking @jjamroga's comments one step further. Can we actually stage/stack this functionality? This is a very big PR. Can we start with manual revert and then layer auto-revert on top?

[adalundhe]

Taking @jjamroga's comments one step further. Can we actually stage/stack this functionality? This is a very big PR. Can we start with manual revert and then layer auto-revert on top?

^^ I'm switching this back to draft and getting this done today.

[adalundhe]

Split update for the staging request in the review: the implementation now has six open, non-draft PRs. Each dependent PR targets its immediate predecessor.

Part	Scope	PR	Files	Added / removed
1	Durable recovery storage and Substrate primitives	kagent-dev/kagent#3255	6	+438 / -45
2	Manual crashed-runtime workflow and admitted-operation retries	adalundhe/kagent#1	5	+646 / -4
3	Public RevertSession RPC and generated clients	adalundhe/kagent#2	15	+674 / -98
4	Recovery observations and exact discarded-message references	adalundhe/kagent#3	17	+889 / -188
5	Manual chat recovery choices and failed-message indicators	adalundhe/kagent#4	14	+363 / -10
6	Opt-in automatic recovery	adalundhe/kagent#5	35	+650 / -163

Parts 2–6 are dependent review previews in the contributor fork. They need new upstream PRs as their predecessors merge; they are not upstream merge targets yet. The fork cannot supply base branches to this upstream repository.

The split preserves the requested behavior:

• Manual recovery lands before automatic admission. Background retries in the manual layers process only operations already admitted.
• Controller-wide sessions.automaticRecovery defaults to false. Automatic mode shows progress without manual recovery choices.
• Manual chat offers Revert or a fork from an existing READY checkpoint.
• Exact task/message references mark discarded inputs “Failed to send.” Completed A2A tasks and history remain intact.
• Redundant comments were removed. The automatic layer updates every current workflow caller, including the waiting-expiration test behind the earlier type-check failure.

Local validation passes for the affected PostgreSQL/migration and backend race suites, Go lint, protobuf compatibility, UI behavior tests, typecheck, and lint. The UI layers have 82 and 83 passing behavior tests, respectively.

Practical validation now also uses an isolated KIND cluster with Substrate 0.6.0 and controller/Go ADK images built from the stack. Real API/browser checks passed for checkpoint forks, recovery after a lost final Revert response, retained HITL waits, opt-in automatic recovery, and exact failed-message indicators after reload.

Adversarial review is complete for the current six heads. Three reproduced defects still need fixes:

• P1, parts 1–2: abrupt worker failure during the ordinary post-turn idle snapshot leaves its issued claim unresolved. The real API reports recovery.needed=true, but repeated Revert calls return Aborted: resource conflict. Controlled PostgreSQL probes also reproduce the block across workflow restarts and both automation settings.
• P1, parts 1–2: a delayed concurrent Revert can admit an old CRASHED observation after another Revert succeeds and a new user message reaches WORKING. The new task becomes FAILED and the Session remains FAILED/RESUME while the same real Actor stays RUNNING. The existing retained operation generation can fence this stale admission.
• P2, part 5, inherited by part 6: an already-open idle chat refreshes task history without refreshing Session recovery metadata. After a real crash, recovery controls remain absent and the composer stays enabled; reload immediately corrects both. The reproduction uses a supported separate lifecycle client before exclusive worker loss.

All three have real KIND/API evidence, with browser screenshots for the UI case. The concurrency probe delays one genuine Substrate response through a local instance of the production SessionService; it uses the real database, Actor and Go runtime. No persisted state or API response payload is fabricated. The native Claude/Codex runtime matrix was not rerun. No fixes for these findings have been pushed, and the six application worktrees remain unchanged.

Current upstream CI is not green: CI Build and Migration Immutability still report action_required and await maintainer approval. The fork previews target non-main branches, which do not trigger the existing main/release PR workflow filters.

[EItanya]

🤖 AI-generated review.

Reviewed with #3409 landing first and this stack rebased on top.

[EItanya]

Can we capture the current operation generation before reading Substrate and compare it under this lock? Revert A can observe CRASHED, pause while Revert B succeeds and a new turn starts, then pass both this Actor UID check and READY/NONE. It fails the new task before discovering that the Actor is RUNNING, leaving the Session FAILED/RESUME. This remains reachable after #3409 through ordinary ingress wake-up.

[adalundhe]

Done in 7490323. Revert now reads the Session's lifecycle generation, its latest operation_id, before it calls Substrate. BeginCrashedSessionRecovery compares it under the Session lock. A revert that finished in between makes the stale admission return ABORTED, and the new turn keeps running. TestCrashedRecoveryAdmissionRespectsFences covers the store, and TestExplicitRevertRejectsCrashObservedBeforeAnotherRevert in #3395 replays your sequence. Without the check, that test fails the new task and then stops on the RUNNING Actor.

[EItanya]

A crash during ordinary post-turn suspension leaves an issued quiescence claim here. quiesceIdleSession retains it on error, ClaimSessionQuiescence cannot reclaim it, and this guard keeps rejecting recovery even after a controller restart. Can we reconcile that issued outcome against the recorded Actor's confirmed crash, then retire the obsolete claim atomically with recovery admission? We still need the exclusion while suspension is genuinely unresolved.

[adalundhe]

Done in 7490323. Once the recorded Actor's crash is confirmed, admission no longer waits on an issued idle suspension, because that suspension cannot finish against a crashed Actor. Settlement retires the claim in the admission transaction. A late FinishSessionQuiescence for it finds no pending work and records nothing. The generation check keeps the crash observation current. The claim still excludes task writes, checkpoints, expiration and lifecycle operations, and checkpoint creation still excludes recovery. TestCrashedRecoveryRetiresIssuedIdleSuspension covers the retirement, the late finish and the next turn.