Repository navigation
ateapi: retry a crashed golden actor up to 3 times - #98
Closed
danielle-mustillo wants to merge 2 commits into
Closed
danielle-mustillo wants to merge 2 commits into
danielle-mustillo wants to merge 2 commits into
Conversation
The template reconciler reverts a CRASHED golden actor and boots it again after a 15s, 1m, then 4m backoff with up to 20% jitter. GoldenSnapshotStatus records the retries in attempts and last_crash_time. After the last retry the template fails as before, with the crash message appended.
A revert keeps failing while the crashed actor's atelet is unreachable. Fail the template 5m after the retry was due. Fail a REVERTING golden actor whose crash the reconciler did not count, as before.
This was referenced Oct 23, 2026
Merged
adalundhe
pushed a commit
to adalundhe/kagent
that referenced
this pull request
Oct 29, 2026
…mplates (kagent-dev#3335) ## Why A golden build can fail for an infrastructure reason, such as a lost worker pod or a full node disk, or for a config fault. Substrate v0.7.0-alpha1 treats every golden error as terminal: once it records `error_message`, it never works on that template again. The Agent or SandboxTemplate stays not Ready until an operator deletes and reapplies it. ## What changed - On any golden error, the Agent and SandboxTemplate reconcilers delete the failed ActorTemplate by UID and recreate it after 15s, 1m, 4m, then every 10m, with up to 20% jitter and no limit. A ready build resets the count. - While a retry waits, Ready=False has the new reason `ActorTemplateRetrying`, e.g. `golden build failed (GoldenActorCrashed), retry 1 at <time>`. Status carries only the reason; the full Substrate error goes to the controller log, because `GoldenActorInvalid` can quote configuration. `ActorTemplateFailed` is no longer produced. - 4 new `runtime_revision` columns hold the retry count, failed UID, retry time and the ActorTemplate spec digest, so a controller restart keeps the schedule. GC deletes a rebuild the controller never recorded only when its digest matches. ## Tradeoffs - No retry limit. A limit makes any outage longer than the backoff terminal again. A config fault retries every 10m until the user edits the resource, and the new revision starts from zero. - Retrying in Substrate instead (kagent-dev/substrate#98, kagent-dev/substrate#99) needs a Substrate release. This works with the version kagent pins today. ## Blast Radius - The delete has a UID precondition, so it cannot remove a template that was already replaced. - The columns are added to `000001_initial.sql` in place, so an existing database must be reinstalled. - No UI change. The Agents page shows a retrying Agent as failed, with the retry in its message. ## Size Against main `a9a53207`: | Part | Files | Added | Removed | |---|---:|---:|---:| | Source | 13 | 371 | 125 | | Tests | 14 | 862 | 102 | | Docs | 2 | 25 | 6 | | **Total** | **29** | **1258** | **233** | ## Verification - `go test` for `internal/controller`, `internal/database`, `internal/substrate` and `pkg/migrations` in `go/core`, with PostgreSQL and envtest, passes. - Kind at `3285acf5`, Agent and SandboxTemplate with a crashed golden build: retry 1, a new template UID, then Ready. Substrate's raw error text appears in status 0 times. The key-free suite passes 37 of 37. - Kind, Agent whose image digest was pushed only after retry 3: Ready on retry 4 with no manual step. AI use: AI-assisted, I reviewed it. --------- Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A crashed golden actor fails its ActorTemplate for good today, even when the crash came from infrastructure (worker pod deleted, node disk full). This PR makes the template reconciler retry it.
The reconciler reverts a CRASHED golden actor with
RevertActorand boots it again, after a 15s, 1m, then 4m backoff with up to 20% jitter. It counts each crash inGoldenSnapshotStatusbefore it acts, so a reconciler that dies halfway cannot boot the actor without counting it. After the last retry the template fails as before, with the crash message appended. If the revert keeps failing (for example, the crashed actor's atelet is down), the template fails 5 min after the retry was due.API change.
GoldenSnapshotStatusgetsattempts = 4andlast_crash_time = 5. Old clients skip the new fields.Trade-off. Every client now waits about 5.5 min for a config fault (bad digest, bad entrypoint) to fail, instead of seconds.
Prototype, one of three candidate fixes. The kagent-only alternative is kagent-dev/kagent#3335. It clashes with the S3 + W2 prototype (#99), which uses field numbers 4 and 5 for other fields. Only one of the two can land as written.
AI use: AI-assisted, I reviewed it.