fix: export synthetic OpenTelemetry roots

otel-plugin/README.md

@@ -72,6 +72,13 @@ Unowned ordinary helper failures retain their observation-only behavior. Direct
 worker. Observation cancellation cannot skip the actual task or discard its operation failure. This boundary does
 not reclassify handler/predicate failures or add general wrapped-fatal classification.
 
+Successful-output preparation closes new checkpoint-continuation admission before serialization. This does not
+revoke a user worker already admitted to the configured executor: such a worker can begin after invocation End,
+and already-running unawaited work can also finish afterward. End therefore does not guarantee that all operation
+work or its telemetry is complete. Caller-side manager cleanup retains the core lifecycle above. The remaining
+worker-start and mandatory large-result checkpoint acknowledgment boundaries are tracked in
+[issue #787](https://github.com/aws/aws-durable-execution-sdk-java/issues/787).
+
 SDK inspection of MDC-capture failures reads each visited standard transport cause once and detects identity cycles.
 Cyclic, null, or unreadable leading `CompletionException` chains retain the original wrapper; ordinary initialization
 still reports `FAILED` when its error response can be serialized. This provides cycle safety for finite cause graphs,
@@ -113,6 +120,52 @@ If you configure your own `SdkTracerProviderBuilder`, add the OpenTelemetry SDK
 </dependency>
 ```
 
+## Fallback execution roots
+
+When the backend supplies no complete remote parent and the installed SDK sampler selects `RECORD_AND_SAMPLE`,
+both views export a `DurableExecutionRoot` anchor during the invocation Start hook, including invocations that later
+suspend with `PENDING` or fail with `RETRYING`. The anchor starts before descendants and ends immediately with the
+same checkpointed execution-start timestamp. It is zero-duration: descendants can end after it, and its ended span
+context remains a valid parent. Trace viewers that enforce parent/child time containment may flag this shape. For a
+deferred sampler, the first actual fallback span is now `DurableExecutionRoot`; custom policies that depend on the
+first span name or attributes can observe that ordering. No arbitrary name/input invariance is promised.
+
+If result serialization or oversized-result checkpointing fails before the SDK output is prepared, End reports
+`RETRYING` on the handler worker before the original preparation failure propagates. End flushes completed
+spans without changing handler failure classification or claiming coverage of later response-stream writes or runtime
+acknowledgment. A root end processor's `Exception` or `LinkageError` is logged without suppressing descendant telemetry
+or the End flush. Other `Error` failures still escape the Start hook; if invocation cleanup does not run, a batch
+processor may not flush. A failing processor need not have exported the root itself. Existing End cleanup failure
+priority remains unchanged.
+
+The anchor is marked `durable.execution.synthetic_root=true`. Its trace and span IDs are deterministic and its start
+and end timestamps are the checkpointed execution start. It does not report execution status or duration; `Workflow`
+continues to report those at terminal completion. With the SDK sampler installed, the anchor carries only the two SDK
+identity attributes and the sampled decision, excluding sampler-added attributes and updated trace state. `RECORD_ONLY`
+and `DROP` do not export an anchor; descendants retain their full sampling result, including `RECORD_ONLY`, attributes,
+and trace state. Complete remote parents remain externally owned and are never exported.
+Java's `InvocationInfo` requires a non-null execution start timestamp from the initial checkpointed execution operation.
+Missing timestamps are rejected before the plugin runs; anchor timestamps never fall back to the current wall clock.
+
+Every invocation may re-export the anchor to recover from an earlier interrupted or lost export. Re-exports retain the
+same span fields under stable sampling. The existing provider resource still applies: if a later invocation runs in another
+execution environment, resource attributes such as `faas.instance` can differ. Backends that deduplicate by span identity
+may retain either copy's resource. Each execution ARN owns its own anchor even when multiple executions share a
+propagated trace ID without a parent; they are not collapsed into one
+execution. With the SDK sampler retained in the provider, upstream sampling and configured fallback sampling apply
+to anchors and their descendants together.
+
+After the first invocation's export and flush succeed, its anchor remains available even if the execution is stopped
+or times out while suspended and never invokes the plugin again. Ending the anchor during Start also lets batch
+processors export it while customer code runs. An invocation killed before an export or flush can still lose spans;
+a later sampled invocation, including terminal completion, attempts to export the anchor again. Export
+and flush failures do not provide a delivery guarantee.
+
+For a consistent hierarchy across invocations, preserve an explicit upstream sampling decision or use a deterministic
+sampling policy based on the stable trace ID. A non-deterministic sampler can export an anchor without a Workflow, or
+a Workflow without its anchor. Sampler metadata on descendants remains provider-owned; it does not change the SDK
+anchor's body. Provider resources and custom span processors can still change exported data.
+
 ## Choose one durable OTel view
 
 Configure exactly one of `InvocationOtelPlugin` or `ExecutionOtelPlugin` when enabling durable tracing.
@@ -308,8 +361,9 @@ The SDK-wide sampling guarantees below require `DurableSampler` itself to be the
 [builder constructors](#configuration) and [ADOT extension setup](#1-adot-lambda-layer) install it automatically;
 configure its delegate through these documented paths and retain `DurableSampler` as the final sampler. An
 unrecognized outer wrapper is treated as a plain replacement. The SDK wrapper applies its sampling intent to every durable
-span (Workflow, Invocation, operation and attempt), preserving the full result, including `RECORD_ONLY`, sampler attributes
-and trace state.
+span. Workflow, Invocation, operation and attempt spans preserve the full result, including `RECORD_ONLY`, sampler
+attributes and trace state. The re-exported `DurableExecutionRoot` is the narrow exception described above: it uses only
+the sampled decision and stable SDK identity, and is omitted for `RECORD_ONLY` or `DROP`.
 
 The supported SDK sampler follows this precedence, highest first:
 
@@ -334,6 +388,14 @@ For explicit execution-level control, keep `DurableSampler` as the final install
 for example through X-Ray
 active tracing. The SDK sampling wrapper preserves that decision.
 
+Parentless fallback-root exports also use the provider's sampler. If the SDK wrapper is replaced by a plain
+sampler, those anchors can follow its normal root policy independently of the flags used by their descendants;
+full upstream sampling overrides require the installed SDK wrapper. For example, a sampler that selects spans by name
+can export only the anchor or only its Invocation child. This behavior is covered for both views by
+`ExecutionRootParityTest.plainReplacementRetainsIndependentPerSpanSampling`; installing `DurableSampler` restores the
+shared decision. Merely delaying anchor creation until terminal completion would not make an arbitrary replacement
+sampler consistent across spans.
+
 ## Span Attributes
 
 ### Invocation Span

otel-plugin/src/main/java/software/amazon/lambda/durable/otel/DurableExecutionRoot.java

@@ -0,0 +1,67 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+package software.amazon.lambda.durable.otel;
+
+import static software.amazon.lambda.durable.otel.SpanAttributes.DURABLE_EXECUTION_ARN;
+import static software.amazon.lambda.durable.otel.SpanAttributes.DURABLE_EXECUTION_SYNTHETIC_ROOT;
+
+import io.opentelemetry.api.trace.Span;
+import io.opentelemetry.api.trace.SpanContext;
+import io.opentelemetry.api.trace.SpanKind;
+import io.opentelemetry.api.trace.Tracer;
+import io.opentelemetry.context.Context;
+import java.time.Instant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Creates the SDK-owned zero-duration identity anchor for a fallback execution trace. */
+final class DurableExecutionRoot {
+    private static final Logger logger = LoggerFactory.getLogger(DurableExecutionRoot.class);
+
+    private DurableExecutionRoot() {}
+
+    /** Ends the anchor before customer code without letting an isolated processor failure suppress its descendants. */
+    static void end(Span root, Instant timestamp) {
+        if (root == null) return;
+        try {
+            root.end(timestamp);
+        } catch (Exception | LinkageError failure) {
+            logger.warn(
+                    "Failed to end the execution root; continuing invocation telemetry and flushing at End", failure);
+        }
+    }
+
+    /**
+     * Starts the SDK-owned fallback ancestor before descendants so they inherit the provider's actual sampling
+     * metadata. The caller ends this identity anchor immediately at the checkpointed start timestamp. Recovery
+     * re-exports retain identity and timestamps; Workflow carries duration and outcome. Remote parents are never owned.
+     */
+    static Span start(
+            Tracer tracer,
+            DeterministicIdGenerator idGenerator,
+            SpanContext ancestor,
+            String arn,
+            Instant start,
+            DurableSamplingDecision.Intent intent,
+            boolean useContextCarrier,
+            boolean useThreadCarrier) {
+        if (ancestor == null || ancestor.isRemote()) {
+            return null;
+        }
+        var builder = tracer.spanBuilder("DurableExecutionRoot")
+                .setSpanKind(SpanKind.INTERNAL)
+                .setParent(useContextCarrier ? DurableSamplingDecision.store(Context.root(), intent) : Context.root())
+                .setAttribute(DURABLE_EXECUTION_ARN, arn)
+                .setAttribute(DURABLE_EXECUTION_SYNTHETIC_ROOT, true)
+                .setStartTimestamp(start);
+        Span root;
+        if (useThreadCarrier) {
+            try (var ignored = DurableSamplingDecision.openScope(intent)) {
+                root = idGenerator.startSpan(builder, ancestor.getTraceId(), ancestor.getSpanId());
+            }
+        } else {
+            root = idGenerator.startSpan(builder, ancestor.getTraceId(), ancestor.getSpanId());
+        }
+        return root;
+    }
+}

otel-plugin/src/main/java/software/amazon/lambda/durable/otel/DurableSampler.java

@@ -8,6 +8,7 @@
 import io.opentelemetry.sdk.trace.SdkTracerProviderBuilder;
 import io.opentelemetry.sdk.trace.data.LinkData;
 import io.opentelemetry.sdk.trace.samplers.Sampler;
+import io.opentelemetry.sdk.trace.samplers.SamplingDecision;
 import io.opentelemetry.sdk.trace.samplers.SamplingResult;
 import java.util.Collections;
 import java.util.LinkedHashMap;
@@ -21,7 +22,7 @@
  * <p>The durable plugins decide sampling for the whole execution exactly once per invocation and attach the resulting
  * {@link SamplingResult} to the {@link Context} used as the parent of each durable span (via
  * {@link DurableSamplingDecision}). When {@code shouldSample} sees that decision on the parent context it returns it
- * verbatim, so:
+ * verbatim for descendants. Re-exported execution anchors use only the sampled decision and stable SDK identity, so:
  *
  * <ul>
  *   <li>the wrapped (customer-configured or ADOT/community default) sampler is invoked at most once per invocation for
@@ -120,17 +121,22 @@ public SamplingResult shouldSample(
             // Not a durable span: the customer's sampler governs it unchanged.
             return delegate.shouldSample(parentContext, traceId, name, spanKind, attributes, parentLinks);
         }
-        if (!intent.isDeferred()) {
-            // A resolved decision (explicit upstream, same-trace ambient, or a locally reproduced sampler): return it
-            // verbatim, preserving RECORD_ONLY and never re-invoking the delegate.
-            return intent.resolved();
+        var result = intent.isDeferred()
+                ? deferredDecisions.computeIfAbsent(
+                        new ExecutionKey(
+                                intent.deferredTraceId(), attributes.get(SpanAttributes.DURABLE_EXECUTION_ARN)),
+                        key -> delegate.shouldSample(
+                                Context.root(), key.traceId(), name, spanKind, attributes, Collections.emptyList()))
+                : intent.resolved();
+        if (name.equals("DurableExecutionRoot")
+                && Boolean.TRUE.equals(attributes.get(SpanAttributes.DURABLE_EXECUTION_SYNTHETIC_ROOT))) {
+            // Re-exported anchors carry only SDK identity and the sampled decision. Per-invocation sampler
+            // metadata remains on descendants; recording-only executions do not materialize an anchor.
+            return result.getDecision() == SamplingDecision.RECORD_AND_SAMPLE
+                    ? SamplingResult.recordAndSample()
+                    : SamplingResult.drop();
         }
-        // Deferred (agent path, real sampler not reproducible here): evaluate the actual delegate once per execution
-        // and reuse it, so an installed drop/rate-limit policy is honored and consulted only once.
-        return deferredDecisions.computeIfAbsent(
-                new ExecutionKey(intent.deferredTraceId(), attributes.get(SpanAttributes.DURABLE_EXECUTION_ARN)),
-                key -> delegate.shouldSample(
-                        Context.root(), key.traceId(), name, spanKind, attributes, Collections.emptyList()));
+        return result;
     }
 
     @Override

otel-plugin/src/main/java/software/amazon/lambda/durable/otel/DurableSamplingDecision.java

@@ -100,20 +100,6 @@ static Scope openScope(Intent intent) {
         };
     }
 
-    /**
-     * Inspects the unconsumed durable sampling intent for a span, or {@code null} when none is present. Prefers the
-     * full-fidelity context key (same class loader) and falls back to the thread-scoped system property (cross class
-     * loader).
-     */
-    static Intent get(Context context) {
-        var holder = context.get(KEY);
-        var fromContext = holder != null ? holder.get() : null;
-        if (fromContext != null) {
-            return fromContext;
-        }
-        return fromScopedProperty();
-    }
-
     /** Consumes both carriers before callbacks can reuse the parent context or cross-loader bridge. */
     static Intent consume(Context context) {
         try {