Skip to content

[Bug]: Virtual child contexts exit replay early and omit resumed operation notifications #889

Open

Description

@zhongkechen

Expected Behavior

Entering run_in_child_context(..., config=ChildConfig(is_virtual=True)) during replay must not treat the absence of the virtual container's checkpoint as evidence of new work. A virtual container intentionally has no checkpoint, while its inner operations can have persisted history.

When the next inner operation is an existing successful wait reported as updated since the previous invocation, the child should retain replay status long enough to consume that history and deliver the normal operation-update notification. It can then transition to NEW at the actual replay boundary.

Actual Behavior

The core SDK changes the parent from REPLAY to NEW before creating the virtual child. The child inherits NEW, so its existing wait bypasses the replay/update notification gate.

A core-only comparison, with the same completed wait in the supplied state, produces:

{'virtual': False, 'child_enters_replay': True, 'wait_update_hooks': ['virtual-wait'], 'result': 'ok'}
{'virtual': True, 'child_enters_replay': False, 'wait_update_hooks': [], 'result': 'ok'}

Both calls return the stored successful wait outcome. The demonstrated defect is replay-state classification and the omitted notification; this reproduction does not demonstrate a failed wait or repeated business side effects. An observable consequence is missing terminal wait telemetry in Execution view when a virtual child suspends and resumes. Other consumers of replay status, including replay-aware logging, may also be affected, but that broader impact has not been validated here.

Steps to Reproduce

The motivating workflow is a virtual child running step -> durable wait (5 s) -> step, with the wait completing between invocations. On resume, the virtual container has no checkpoint by design, but its inner operations do.

The following focused reproduction isolates the core boundary without AWS credentials, an OTel plugin, or an exporter. It uses the real DurableContext.run_in_child_context and wait implementations, with a mocked execution-state boundary that supplies a successful updated wait. The ordinary-child control has a STARTED container checkpoint; the virtual child correctly has none.

Run from the Python SDK repository root at the tested revision:

PYTHONDONTWRITEBYTECODE=1 hatch run dev-otel:python - <<'PYTHON'
from datetime import UTC, datetime
from unittest.mock import Mock
from aws_durable_execution_sdk_python.context import DurableContext, ExecutionContext
from aws_durable_execution_sdk_python.config import Duration, ChildConfig
from aws_durable_execution_sdk_python.state import ReplayStatus, CheckpointedResult
from aws_durable_execution_sdk_python.lambda_service import (
    Operation, OperationType, OperationStatus, OperationSubType,
)

for virtual in (False, True):
    state = Mock()
    state.wrap_user_function.side_effect = lambda func, *args, **kwargs: func
    state.durable_execution_arn = 'test-arn'
    parent = DurableContext(
        state, ExecutionContext('test-arn'), replay_status=ReplayStatus.REPLAY
    )
    context_id = parent._peek_next_operation_id()
    child_probe = parent.create_child_context(context_id, is_virtual=virtual)
    wait_id = child_probe._peek_next_operation_id()
    timestamp = datetime(2026, 10, 30, tzinfo=UTC)
    operations = {
        wait_id: Operation(
            operation_id=wait_id, operation_type=OperationType.WAIT,
            status=OperationStatus.SUCCEEDED, name='virtual-wait',
            parent_id=None if virtual else context_id,
            sub_type=OperationSubType.WAIT,
            start_timestamp=timestamp, end_timestamp=timestamp,
        )
    }
    if not virtual:
        operations[context_id] = Operation(
            operation_id=context_id, operation_type=OperationType.CONTEXT,
            status=OperationStatus.STARTED, name='virtual-context',
            parent_id=None, sub_type=OperationSubType.RUN_IN_CHILD_CONTEXT,
            start_timestamp=timestamp,
        )
    state.get_checkpoint_result.side_effect = lambda op_id: (
        CheckpointedResult.create_from_operation(operations[op_id])
        if op_id in operations else CheckpointedResult()
    )
    state.is_operation_updated_since_last_invocation.side_effect = (
        lambda op_id: op_id == wait_id
    )
    child_states = []
    def body(child):
        child_states.append(child.is_replaying())
        child.wait(Duration.from_seconds(5), name='virtual-wait')
        return 'ok'
    result = parent.run_in_child_context(
        body, name='virtual-context', config=ChildConfig(is_virtual=virtual)
    )
    emitted_wait = [
        call.args[0].name
        for call in state.emit_operation_update_hook.call_args_list
    ]
    print({
        'virtual': virtual,
        'child_enters_replay': child_states[0],
        'wait_update_hooks': emitted_wait,
        'result': result,
    })
PYTHON

dev-otel is only the available development environment used to run this probe; the script imports core SDK modules and the Python standard library only. This is an isolated core reproduction, not a new cloud or end-to-end OTel validation.

SDK Version

Runtime reproduction: source checkout 44b9f4d1c862b50041423e19fe6a250bbf7bd72f, whose core package declares 3.0.0.

The same checkpoint-existence predicate and child replay-state inheritance remain in current main 4625d01bbfa97da67050014851af7eb41c61a842, as checked by source comparison on 2026-10-30. The runtime probe above was executed at the first revision; it does not claim a full runtime validation of the later revision.

Python Version

3.15.2, using the existing Hatch development environment on macOS. Other Python versions were not rerun for this report.

Is this a regression?

Unknown; no last-known-working version has been established.

Additional Context

The source path at the reproduced revision is:

  1. _replay_aware, context.py:600: while replaying, not next_exists immediately calls _set_replay_status_new(), including for a virtual container.
  2. run_in_child_context, context.py:833: the child is created inside that replay-aware scope, after the transition.
  3. create_child_context, context.py:444: the child inherits its creator's replay status.
  4. The operation-update/replay gate, context.py:564, runs only while replaying. The inner wait bypasses it and returns its stored success.

Suggested acceptance criteria:

  • Preserve replay state through virtual-container entry when its inner history still needs replay; switch to NEW when actual new work is reached.
  • Exercise initial invocation and resume with step -> wait -> step, checking that completed step bodies do not rerun and the updated wait's normal core notification is delivered once.
  • Cover nested virtual contexts, ordinary-child controls, and a virtual context whose next inner operation is genuinely new.
  • Preserve virtual checkpoint semantics, stable operation IDs, and wire ParentId; do not fabricate a container checkpoint to obtain replay state.
  • Keep rejecting a non-virtual container checkpoint encountered under a virtual child, as required by [Bug]: Virtual child replay can consume a non-virtual container checkpoint #704.

Related but separate:

This issue is scoped to the Python core replay boundary. Virtual-span ancestry, cross-invocation start-time reconstruction, and execution-wide virtual-span export deduplication should be tracked separately.

Activity

  1. hln33 commented on Oct 30, 2026

    @hln33
    Contributor

    Severity - medium

    What is wrong. Entering a virtual child context during replay flips the context to NEW before the child body runs. _replay_aware treats the missing container checkpoint as new work (context.py:612–613). The child is created inside that scope (context.py:859–871) and inherits NEW (context.py:446–448). Inside the child, is_replaying() returns False while history is still being consumed. The resumed wait therefore skips the replay/update hook gate (context.py:576–594). That gate is the only path that delivers on_operation_end for an operation that completed during suspension (state.py:688–699 → plugin.py:1850). So plugins never receive on_operation_end for that wait, and in the OTel Execution view the wait span is never exported. The durable logger also stops de-duplicating (logger.py:153), so log lines inside the virtual child that were written in the earlier invocation are written again on resume.

    What is correct. Execution outcome, step skipping, the wait's returned value, checkpoint IDs, and ParentId wiring are unchanged. Completed step bodies do not re-run; that path is checkpoint-driven, not replay-status-driven. Map and parallel virtual branches already keep REPLAY status (executor.py:679–693: the is_virtual branch only validates; the flip at 687–688 is in the elif). So the correct rule exists in the codebase.

    Why durable work is not re-executed. The replay flag and the checkpoint lookup are separate mechanisms. Each step and wait looks up its own checkpoint by operation ID (operation/step.py:96–105, operation/wait.py:61–70). If the stored result is SUCCEEDED, the SDK returns it and the body never runs. Nothing under operation/ reads is_replaying() or _replay_status. So the skip decision does not depend on the flag, and this bug cannot cause a step or wait to run twice. The flag has three consumers: the durable logger, the plugin hook gate, and the public is_replaying() method. Those three are what the bug breaks. The only path where user-visible work can repeat is user code placed outside a step and guarded by if not ctx.is_replaying():. Inside a virtual child, that guard is wrong on resume, so such code runs again. Work inside a step is protected regardless.

    Scope. Only users of ChildConfig(is_virtual=True). wait_for_callback passes a non-virtual config (context.py:953–958), and map/parallel use the executor path above. The effect is missing telemetry, duplicated logs, and a wrong is_replaying() value, not a wrong result. Note: _set_replay_status_new is one-way (context.py:501–503), so the parent context also stays NEW after the virtual child returns; later operations in the same invocation lose their replay/update hooks as well. This follows from the code and extends the scope slightly beyond the child body.

    Regression status. Partly a regression. The per-context existence flip arrived with #488 ("context aware replay status", merge 86642948) and first shipped in sdk-v1.7.0. Before that, sdk-v1.6.0 tracked replay status globally on ExecutionState with a visited set (state.py:370–404 at sdk-v1.6.0,otel-v0.2.0). That mechanism stayed REPLAY until every completed operation had been visited. In this scenario the completed inner wait is unvisited when the virtual child body starts, so 1.6.0 kept REPLAY through virtual-child entry. The 1.6.0 logger read that global status (logger.py:131 at the same tag). So the duplicate-log symptom is a regression from 1.6.0 to 1.7.0. The other two symptoms are not regressions: DurableContext.is_replaying() and the plugin replay/update hooks did not exist before 1.7.0. Every release from 1.7.0 through 3.0.1 has the current behavior.

    Why not higher severity. No data loss, no wrong result, no broken build or release. Step bodies and waits are protected by their checkpoints. The affected surface is observability and the is_replaying() contract, on an opt-in feature.

    Why not lower severity. is_replaying() is public API and returns a wrong value. Plugin authors cannot work around a missing on_operation_end. The same construct already behaves correctly on the map/parallel path, so the fix is a known rule applied to one more call site.

    Line references are against main at c7266bba.

  2. added
    pkg:pluginsPlugin interface, possibly impacting all plugins
    and removed on Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugpkg:pluginsPlugin interface, possibly impacting all pluginspkg:sdkPackage: aws-durable-execution-sdk-python

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions