Skip to content

[Feature]: Java Workflow Insight plugin follow-up work #679

Description

@wangyb-A

What would you like?

Add a Workflow Insight plugin to the AWS Lambda Durable Execution SDK for Java.

The plugin should emit a structured summary of each durable execution, including
execution status, timing, input/output, errors, and operation details. It should
follow the JavaScript Workflow Insight contract where practical while accounting
for Java-specific threading, serialization, and plugin lifecycle behavior.

Why?

Customers need an easier way to inspect and analyze durable workflows without
reconstructing execution state from raw Lambda logs and execution history.

Workflow Insight records enable:

  • Execution monitoring and troubleshooting
  • Workflow latency and failure analysis
  • Operation-level visibility
  • Querying records through CloudWatch Logs, S3, and analytics tools
  • Consistent observability across JavaScript, Python, and Java SDKs

Initial scope

  • Add a publishable insight-plugin Maven module.
  • Support schema version 1.0 of the Workflow Insight record.
  • Support ON_COMPLETE, ON_FAILURE, and ON_CHANGE emission modes.
  • Support deterministic per-execution sampling.
  • Support top-level and full-tree operation detail.
  • Support input/output transforms, operation result transforms, filtering, and
    error controls.
  • Support record-size limits and deterministic truncation.
  • Isolate plugin and exporter failures from durable execution.
  • Publish the plugin to Maven Central and attach it to GitHub releases.

Acceptance criteria

  • Java plugin hooks expose the execution and operation data required by the
    record contract.
  • Workflow Insight module builds as part of the Maven reactor.
  • Records include execution metadata, operation summaries, errors, attempts,
    and opted-in results.
  • S3 emits the canonical operations array.
  • Lambda and CloudWatch Logs emit operationsByName.
  • Sampling, content controls, filtering, truncation, and exporter isolation
    have unit coverage.
  • S3 Workflow Insight conformance passes all applicable requirements.
  • CloudWatch Workflow Insight conformance passes all applicable requirements.
  • Release automation publishes the new Maven artifact.
  • Public preview APIs use @Experimental.
  • includeErrors(false) suppresses execution and operation errors.
  • Plugin Throwables do not disrupt durable execution.
  • Add an ADR documenting the record contract, lifecycle, sampling,
    truncation, content-transform, and exporter-ordering decisions.
  • Add customer-facing installation, IAM, configuration, schema, and
    troubleshooting documentation.
  • Add automated cloud integration coverage for both S3 and CloudWatch.
  • Complete final human review and merge the implementation PR.

Design constraints

  • Plugin failures must never change the durable execution result.
  • Remote exporter I/O must not block SDK checkpoint coordination.
  • Memory use must remain bounded across suspended and resumed executions.
  • Terminal records must not be overwritten by delayed RUNNING records.
  • Sensitive input, output, and error content must be configurable.
  • Sampling decisions and record behavior should remain aligned across SDKs.
  • Breaking record-contract changes require a schema-version change.

Follow-up work

Check the comments

Links

Is this a breaking change?

No.

Does this require an RFC?

No.

Activity

  1. self-assigned this
    on Sep 2, 2026
  2. added
    parityProvides parity with other language implementations of the SDK
    on Sep 2, 2026
  3. wangyb-A commented on Sep 8, 2026

    @wangyb-A
    ContributorAuthor

    What would you like?

    Track work deferred from the initial Java Workflow Insight plugin pull request,
    #661.

    Follow-up items

    • Move ON_CHANGE exports off the checkpoint callback thread, add bounded
      coalescing, and prevent a delayed RUNNING export from overwriting the
      terminal object.
      (comment)
    • Preserve Java runtime types when passing values to content transforms.
      (comment)
    • Guarantee that records satisfy the configured size limit after all
      truncation phases. (comment)
    • Distinguish an included JSON null value from an omitted or disabled field.
      (comment)
    • Complete exporter isolation for enum values and mutable map keys.
      (comment)
    • Coordinate sampling normalization across JavaScript, Python, and Java;
      Java should continue matching JavaScript until then.
      (comment)
    • Cloud base integration test
    • Skip input snapshotting when input collection is disabled
      (content.input(false)); cache explicit absence without serializing the value.
      (comment)
    • Validate required exporter configuration, including a non-blank destination
      and positive record-size limit, at build time for S3, CloudWatch Logs, and
      Lambda log exporters.
      (comment)
    • Do not add endTime or durationMs to non-terminal RUNNING records; a
      PENDING or RETRYING invocation boundary is not execution completion.
      (comment)
    • Omit error.message when an exception has no message instead of emitting
      "message": null.
      (comment)
    • Avoid forcing users of the default LambdaLogExporter to include the S3 and
      CloudWatch Logs clients; use optional dependencies or separate exporter
      modules.
      (comment)
    • Avoid re-rendering and serializing the complete record for every truncation
      candidate.
      (comment)
    • Parse durable execution ARNs by structure rather than fixed positions, and
      normalize empty function, qualifier, region, account, and execution names
      consistently.
      (comment)
    • Add public accessors for workflow record input, output, error, end time,
      duration, region, account, and truncation metadata so custom exporters do not
      need to serialize the record to inspect it.
      (comment)
    • Update the plugin README to state that per-invocation state is cleared at every
      invocation end and rebuilt after resume.
      (comment)
    • Move REVIEW_FINDINGS.md out of the Maven-published plugin module; internal
      review-triage notes should remain in the PR or tracking issue.
      (comment)
    • Preserve the Workflow Insight record when an input or output supported by a custom SDK SerDes cannot be copied by the plugin’s Jackson mapper; omit only the unsupported field instead of dropping the entire record.
      (comment)

    Missing exporters

    Java already has Lambda logs, CloudWatch Logs, and S3. Add the remaining
    exporters available in the JavaScript plugin:

    • CloudWatchLogsExporter
    • DynamoDBExporter
    • FirehoseExporter
    • EventBridgeExporter
    • SQSExporter
    • OTelExporter
    • HttpExporter
    • FileExporter
    • OpenSearchExporter
    • RedshiftExporter
    • AuroraExporter

    Links

  4. wangyb-A commented on Sep 9, 2026

    @wangyb-A
    ContributorAuthor

    Draft implementation for the bounded FIFO ON_CHANGE scheduler: #699

  5. wangyb-A commented on Sep 11, 2026

    @wangyb-A
    ContributorAuthor
    • pending slot coalesces across executions, so one execution's final record can be dropped
    Finding 1 (should fix): the pending slot coalesces across executions, so one execution's final record can be dropped
    
       ExportScheduler has one pending field. schedule() overwrites it unconditionally. The coalescing rule is "a newer record supersedes an older one because each record is a complete snapshot of the execution." That rule holds only for records of
       the same execution. A record for execution B does not contain execution A's information. So when records for two ARNs interleave, pending discards a record that nothing else will deliver.
    
       The end-of-invocation path makes this a data-loss bug, not a lost intermediate:
    
       1. Execution A's onInvocationEnd calls closeAndSchedule, which puts A's SUCCEEDED record in pending.
       2. A's hook thread enters drain() and waits on the in-flight pump.
       3. Execution B's onOperationChange calls schedule(), which replaces pending with B's RUNNING record.
       4. The pump takes pending, exports B's record, sees pending == null, and exits.
       5. A's drain() returns. A's terminal record was never exported and its state is removed from byArn.
    
  6. wangyb-A commented on Sep 15, 2026

    @wangyb-A
    ContributorAuthor

    Fix for the cross-execution pending-slot coalescing: #716

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestparityProvides parity with other language implementations of the SDKpkg:sdkModule: sdk

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions