Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .changeset/27877-map-parent-resume-refusal.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
'@objectstack/service-automation': patch
'@objectstack/spec': patch
'@objectstack/runtime': patch
---

fix(service-automation)!: a resume of a run parked on a `map` node, addressed by the parent's run id while the in-flight item's child run is still suspended, is refused with a code naming the child run id, instead of completing the parent past an item that never ran (#27877)

Clause-②: yes (narrowing) — `resume(runId)` on a run whose resume chain reaches a `map` link with a still-suspended item child, which answered success before, is refused, with a new wire code `RESUME_CHILD_PENDING` (a new member of the `AutomationResult.code` union, the `TriggerFlowResponseSchema` `code` enum and the ADR-0112 ledger).
Metadata-protocol: none

<!-- adr-0087: not-required (no-migration-prescription) No authorable key is added, removed, renamed or re-shaped: the one protocol change is a new member of a response code enum, which no metadata author writes. A resume the engine accepted is refused before any state is touched, so there is no stored or authored shape for a migration to rewrite and no tombstone. -->

**BREAKING** accept-set narrowing at `AutomationEngine.resume` (`@objectstack/service-automation`), served by the generic resume route (`POST /automation/:name/runs/:runId/resume`) and the MCP `resume_run` tool (`@objectstack/runtime`), with its code added to the contract (`@objectstack/spec`).

- **Before.** A run parked on a `map` node, resumed by its own id while the item's child run was parked on a pause open to the generic route (a `screen`, a `wait`, any node declaring `resumeAuthority: 'any'`), re-ran the map node. The map had already counted the in-flight item as started, so it moved past it: the resume answered success, the parent's run row was `completed`, and the item child stayed in `listSuspendedRuns()` with nothing left to continue into.
- **Now.** That resume is refused with the new code `RESUME_CHILD_PENDING` (409 on both doors), and the message names the item run id to resume instead. Nothing moves: the parent stays parked on the item, the map's progress is unchanged, and the item child stays resumable. The same holds when a `subflow` link leads down to such a map: the refusal names the item run of the deepest map link.
- **Unchanged.** Resuming the item run continues the map exactly as before, through the engine's own up-bubble. A parent parked on a `subflow` node still delegates the signal to its child. An item parked on a node only its owning service may resume still answers `PERMISSION_DENIED` (403) at the parent's id. A `map` whose item run is gone (cancelled, or never persisted) is not refused.
- **New code.** `RESUME_CHILD_PENDING` joins `AutomationResult.code`, `TriggerFlowResponseSchema.data.code` and the error-code ledger under `@objectstack/service-automation`; the runtime resume door maps it to 409. It is not `RESUME_IN_PROGRESS`: that one is benign and needs nothing from the caller, this one is answered only by resuming a different run id. A client that switches exhaustively over the code union gains one member.
- **Who sees it.** A caller that resumes a batch flow by the parent run id it launched with, while an item is parked on a screen or a wait.

**The remedy.** Resume the item run the refusal names (also listed by `listSuspendedRuns()`, the run whose context carries the parent's id); the parent continues when the map's last item completes.
3 changes: 2 additions & 1 deletion content/docs/references/api/automation-api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ const result = AutomationApiErrorCode.parse(data);

| Property | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +352 more>` | ✅ | The inner refusal's code in its WIRE spelling - the ADR-0112 member a record save of the same write answers in `error.code` (e.g. `UNIQUE_VIOLATION`); the closed set `ApiErrorSchema.code` parses against |
| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +353 more>` | ✅ | The inner refusal's code in its WIRE spelling - the ADR-0112 member a record save of the same write answers in `error.code` (e.g. `UNIQUE_VIOLATION`); the closed set `ApiErrorSchema.code` parses against |
| **message** | `string` | ✅ | The inner refusal's own diagnostic text, verbatim, without the node and operation prefixes the run's failure text adds |

### Allowed Values: `FlowInnerRefusal.code`
Expand Down Expand Up @@ -631,6 +631,7 @@ const result = AutomationApiErrorCode.parse(data);
* `REQUEST_NOT_FOUND`
* `RESEED_NO_ROWS`
* `RESEED_SKIPPED`
* `RESUME_CHILD_PENDING`
* `RESUME_FAILED`
* `RESUME_IN_PROGRESS`
* `RESUME_TARGET_LOST`
Expand Down
3 changes: 2 additions & 1 deletion content/docs/references/api/contract.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ const result = ApiErrorSchema.parse(data);

| Property | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +352 more>` | ✅ | Error code (e.g. VALIDATION_ERROR; StandardErrorCode ∪ the ledger the serving side registers — ERROR_CODE_LEDGER for framework packages) |
| **code** | `Enum<'VALIDATION_ERROR' \| 'INVALID_FIELD' \| 'MISSING_REQUIRED_FIELD' \| 'INVALID_FORMAT' \| 'VALUE_TOO_LONG' \| 'VALUE_TOO_SHORT' \| 'VALUE_OUT_OF_RANGE' \| … +353 more>` | ✅ | Error code (e.g. VALIDATION_ERROR; StandardErrorCode ∪ the ledger the serving side registers — ERROR_CODE_LEDGER for framework packages) |
| **declaredCode** | `string` | optional | The producer-declared code, verbatim, when it is not a member of the closed `code` vocabulary — the open, author-authored channel (app-specific spellings; ADR-0112) |
| **message** | `string` | ✅ | Readable error message |
| **userMessage** | `string` | optional | Producer-marked user-facing refusal text, verbatim. Present exactly when the producer opted in at throw time; consumers render it to end users and keep their generic substitution for anything unmarked. Status-agnostic; never replaces `message`. |
Expand Down Expand Up @@ -319,6 +319,7 @@ const result = ApiErrorSchema.parse(data);
* `REQUEST_NOT_FOUND`
* `RESEED_NO_ROWS`
* `RESEED_SKIPPED`
* `RESUME_CHILD_PENDING`
* `RESUME_FAILED`
* `RESUME_IN_PROGRESS`
* `RESUME_TARGET_LOST`
Expand Down
1 change: 1 addition & 0 deletions content/docs/references/api/error-code-ledger.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -486,6 +486,7 @@ const result = ErrorCode.parse(data);
* `REQUEST_NOT_FOUND`
* `RESEED_NO_ROWS`
* `RESEED_SKIPPED`
* `RESUME_CHILD_PENDING`
* `RESUME_FAILED`
* `RESUME_IN_PROGRESS`
* `RESUME_TARGET_LOST`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,13 +58,14 @@ function makeDispatcher(resumeResult: AutomationResult) {
const ROWS = [...RESUME_REFUSAL_ROWS].map(([code, row]) => [code, row.status] as const);

describe('#21724 — the resume-refusal rows', () => {
it('the table is the six engine refusals the contract publishes', () => {
it('the table is the seven engine refusals the contract publishes', () => {
// A floor, so an emptied or truncated table cannot pass the per-row
// pins below vacuously.
expect(ROWS.map(([code]) => code).sort()).toEqual([
'INVALID_SCREEN_INPUT',
'INVALID_SIGNAL',
'PERMISSION_DENIED',
'RESUME_CHILD_PENDING',
'RESUME_IN_PROGRESS',
'RUN_NOT_FOUND',
'STORE_UNAVAILABLE',
Expand Down
7 changes: 6 additions & 1 deletion packages/runtime/src/domains/automation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2612,6 +2612,7 @@ export const RESUME_REFUSAL_ROWS: ReadonlyMap<string, { status: number; fallback
['RUN_NOT_FOUND', { status: 404, fallback: 'No such suspended run' }],
['STORE_UNAVAILABLE', { status: 503, fallback: 'Suspended-run store unavailable' }],
['RESUME_IN_PROGRESS', { status: 409, fallback: 'Run is already being resumed' }],
['RESUME_CHILD_PENDING', { status: 409, fallback: 'Resume the pending child run instead' }],
]);

/**
Expand All @@ -2625,7 +2626,7 @@ export const RESUME_REFUSAL_ROWS: ReadonlyMap<string, { status: number; fallback
* message on either door. This used to be inline in the REST arm. It moved here
* unchanged, and the arm's wire answers are byte-identical.
*
* The six coded rows are REFUSALS the engine made BEFORE consuming the
* The seven coded rows are REFUSALS the engine made BEFORE consuming the
* suspension, so the run is still parked and the caller can retry (the codes
* and why each has its status are listed at the REST arm).
*
Expand Down Expand Up @@ -3486,6 +3487,10 @@ export async function handleAutomationRequest(deps: DomainHandlerDeps, path: str
// existence is unknown; the same call is expected
// to work once it recovers (#4420)
// RESUME_IN_PROGRESS → 409, a concurrent resume already has this run
// RESUME_CHILD_PENDING → 409, the run waits on a `map` item whose
// child run is still suspended; the engine's
// message names that child run id to resume
// instead, and nothing moved (#27877)
// A result with NO code and `success: false` consumed its pause and
// ran: 400 `FLOW_FAILED` (#8684), whose details carry the engine's
// own verdict since #15221 — `status: 'stranded'` + `repairable`
Expand Down
1 change: 1 addition & 0 deletions packages/runtime/src/mcp-resume-run.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -492,6 +492,7 @@ describe('MCP resumeRun — (d) the engine\'s answers match the REST resume door
['RUN_NOT_FOUND', { success: false, code: 'RUN_NOT_FOUND', error: "Suspended node 'collect' no longer exists" }, 'RUN_NOT_FOUND'],
['STORE_UNAVAILABLE', { success: false, code: 'STORE_UNAVAILABLE' }, 'STORE_UNAVAILABLE'],
['RESUME_IN_PROGRESS', { success: false, code: 'RESUME_IN_PROGRESS', error: 'Run is already being resumed' }, 'RESUME_IN_PROGRESS'],
['RESUME_CHILD_PENDING', { success: false, code: 'RESUME_CHILD_PENDING', error: "Resume run 'run_child' instead" }, 'RESUME_CHILD_PENDING'],
['FLOW_FAILED (stranded)', {
success: false, error: 'tail blew up', status: 'stranded',
errorMessage: 'Please contact support', summary: { nodes: [] },
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
// And it is live during boot, not only in theory: `plugin.ts` `start()` calls
// `rearmSuspendedWaitTimers`, which calls `engine.resume(run.runId)` for every
// overdue run, and the public `resume()` runs its authorization gate
// (`refuseGatedResume` → `resolveEffectiveSuspension`) through THIS degrading
// (`refuseGatedResume` → `resolveSuspensionChain`) through THIS degrading
// reader before `resumeInternal` ever takes the strict one.
//
// ## The fix, and the one thing deliberately NOT changed
Expand Down Expand Up @@ -356,7 +356,7 @@ describe('#6230 — one "store unreachable while resuming" incident, both record
* unrelated finding. `resume()` walks BOTH catches around
* `loadSuspendedRunStrict`:
*
* gate → `refuseGatedResume` → `resolveEffectiveSuspension` →
* gate → `refuseGatedResume` → `resolveSuspensionChain` →
* `loadSuspendedRun` → this seam, `warn` → **stdout**
* then → `resumeInternal` → strict read → #5912, `error` → stderr
*
Expand Down
65 changes: 46 additions & 19 deletions packages/services/service-automation/src/engine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6373,6 +6373,13 @@ export class AutomationEngine implements IAutomationService {
* (subflow delegation / up-bubble, `map` re-entry, wait-timer wake) go
* through {@link resumeInternal} and are not re-gated — they continue work
* some already-authorized call started.
*
* **A map whose item is still parked is refused, not advanced (#27877).**
* Resuming a run parked on a `map` node by ITS OWN id re-runs the map,
* which advances past the in-flight item — so a run whose item child is
* still suspended answers a coded refusal naming that child's run id, and
* nothing moves. Resuming the child is the door; the map continues when it
* completes. See {@link refuseGatedResume}.
*/
resume(runId: string, signal?: ResumeSignal): Promise<AutomationResult> {
// The body lives in ./run-resume.ts (#25222).
Expand All @@ -6398,8 +6405,18 @@ export class AutomationEngine implements IAutomationService {
* chain and judges the node the signal lands on (subflow) or would advance
* past (map) — see {@link LINKED_RUN_PREFIXES}. A run it cannot resolve at
* all (unknown run id, nothing suspended, no resolvable node type) is left to
* `resumeInternal`, which reports the machine-state error — the gate only
* ever speaks to authorization.
* `resumeInternal`, which reports the machine-state error — the gate speaks
* to authorization, and to the one machine-state fact only its chain walk
* can see.
*
* **That fact is a skip (#27877).** When the chain passes a `map` link
* whose item child is still suspended, continuing the id the caller holds
* would re-run the map and advance it PAST that item without running it —
* the parent completes and the item's suspension is left behind, orphaned.
* Authority is judged first, so a service-owned item keeps its 403; an item
* open to this route is refused with a coded 409 naming the child run id to
* resume instead, before any state is touched. A `subflow` link is not a
* skip — the signal is delegated to the child — and is unchanged.
*
* **An UNDECLARED node type is refused, not deferred** (#5561 step two). It
* used to be let through on the schema default's inherited `'any'`; a pause
Expand All @@ -6416,7 +6433,7 @@ export class AutomationEngine implements IAutomationService {
private refuseGatedResume(runId: string, signal?: ResumeSignal): Promise<AutomationResult | null> {
// The body lives in ./run-resume.ts (#25222).
const host: ThisParameterType<typeof refuseGatedResume> = {
resolveEffectiveSuspension: (...a) => this.resolveEffectiveSuspension(...a),
resolveSuspensionChain: (...a) => this.resolveSuspensionChain(...a),
resolveSuspendedNodeType: (...a) => this.resolveSuspendedNodeType(...a),
resolveResumeAuthority: (...a) => this.resolveResumeAuthority(...a),
resolveDeclaredResumeAuthority: (...a) => this.resolveDeclaredResumeAuthority(...a),
Expand Down Expand Up @@ -6594,38 +6611,48 @@ export class AutomationEngine implements IAutomationService {
* node is literally where it lands.
* - `map:` — the signal is not delegated; the `map` node RE-RUNS, and since
* `$mapState.started` was advanced past the in-flight item before the
* suspend, continuing here advances the map *past* the item whose child
* is still parked. Judging the parent's own `map` node (always
* suspend, continuing here would advance the map *past* the item whose
* child is still parked. Judging the parent's own `map` node (always
* `resumeAuthority: 'any'`) let a raw resume skip a pending approval in a
* batch-approval flow — the gate has to read the item, not the loop.
* batch-approval flow — the gate has to read the item, not the loop. And
* since #27877 the gate does not let that advance happen at all: a
* resume that reaches a `map:` link whose child is still suspended is
* refused with a code naming the child run id, whatever the item's
* authority — so no parent completes past an item that never ran, and no
* item suspension is left orphaned. The engine's own up-bubble from the
* completed child goes through `resumeInternal` and is not gated.
*/
private static readonly LINKED_RUN_PREFIXES = ['subflow:', 'map:'] as const;

/**
* Follow the linked-run chain from `runId` to the suspension whose node
* actually governs a resume — see {@link LINKED_RUN_PREFIXES}. The deepest
* reachable suspension, not the id the caller holds, is what a resume
* really continues (or skips past). Returns `null` when nothing is
* suspended under that id; stops at the last resolvable link when a child
* row is gone (that run is where `resumeInternal` will continue).
*/
private async resolveEffectiveSuspension(runId: string): Promise<SuspendedRun | null> {
* actually governs a resume — see {@link LINKED_RUN_PREFIXES}. Returns every
* suspension the walk loaded, the caller's own first: the LAST one, not the
* id the caller holds, is what a resume really continues (or skips past),
* and each `map:` link with a successor is an item still parked under it
* (#27877). Empty when nothing is suspended under that id; stops at the
* last resolvable link when a child row is gone (that run is where
* `resumeInternal` will continue).
*/
private async resolveSuspensionChain(runId: string): Promise<SuspendedRun[]> {
const seen = new Set<string>();
const chain: SuspendedRun[] = [];
let run = await this.loadSuspendedRun(runId);
for (let depth = 0; run && depth < AutomationEngine.MAX_SUSPENSION_CHAIN_DEPTH; depth++) {
chain.push(run);
const correlation = run.correlation;
const prefix = typeof correlation === 'string'
? AutomationEngine.LINKED_RUN_PREFIXES.find(p => correlation.startsWith(p))
: undefined;
if (!prefix) return run;
if (!prefix) return chain;
const childRunId = correlation!.slice(prefix.length);
if (!childRunId || seen.has(childRunId)) return run;
if (!childRunId || seen.has(childRunId)) return chain;
seen.add(childRunId);
const child = await this.loadSuspendedRun(childRunId);
if (!child) return run;
run = child;
run = await this.loadSuspendedRun(childRunId);
}
return run;
// The depth bound stopped the walk on a run it loaded but never read.
if (run) chain.push(run);
return chain;
}

/** Read a suspended run from the hot cache, falling back to the durable
Expand Down
Loading
Loading