[Bug]: Unresolved Async Callback Keeps Execution Idly Running #734

Expected Behavior

DurableFuture.anyOf(cb1, cb2)with async callbacks should not block the overall execution from completing when the first callback settles, but the second does not. See reproduction example below.

Once one callback succeeds or fails, the other unresolved callback should not keep the Lambda invocation alive or gate the final workflow result. The execution should either complete or suspend itself until all callbacks resolve.

Actual Behavior

After one callback settles, but the other still pending, the Durable Execution is blocked from completing and continuously re-invokes itself until all branches settle.

In the callback repro, cb1 is completed, but the execution keeps re-invoking and timing out while teardown waits for the unresolved cb2 branch:

Waiting for operation to complete before shutting down: d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35

d4735e3a... is the losing branch operation ID. The workflow eventually succeeds only when the losing callback times out. If the losing branch has no timeout, the hang can be unbounded.

This is distinct from #707's original suspension bug.

Steps to Reproduce

1. Deploy a durable handler that creates two unresolved callbacks and returns DurableFuture.anyOf(cb1, cb2).
2. Complete only cb1.
3. Observe that the execution does not complete promptly or suspend. It runs idly until invocation times out, re-invokes after some time, times out again, and repeats.
4. Check Lambda logs and execution history.

Example repro shape:

var f1 = context.waitForCallbackAsync("cb1", String.class, callbackSubmitter, config);
var f2 = context.waitForCallbackAsync("cb2", String.class, callbackSubmitter, config);
return String.valueOf(DurableFuture.anyOf(f1, f2));

Screenshot

Note the repeated invocation timeouts/failures after the first callback resolves:

Still Need to Confirm

• Still need to confirm if the issue is unrelated to DurableFuture.anyOf()
• It could be the case that any pending async callback in the background will prevent the execution from completing or suspending

Java Version

21

Is this a regression?

No

Additional Context

Local triage identified two distinct defects in the anyOf path:

• Defect A: anyOf skipped the durable wait/deregister path and never suspended. This is tracked by [Bug]: DurableFuture.anyOf blocks without deregistering the active thread, so the execution never suspends #707 and addressed by PR fix: DurableFuture.anyOf blocking without deregistering the active thread #730.
• Defect B: anyOf does not cancel or abandon the losing branch, so teardown waits for it in ExecutionManager.validateRunningThreads().

The fix should add regression coverage that asserts:

1. A Durable Execution is not kept idly running when there is an unresolved async callback in the background
2. The losing branch is cancelled, deregistered, or otherwise prevented from blocking teardown. Either that or the Durable Function is suspending until the callback resolves.