Fix workflow test retries and local activity shutdown

[DABH]

What was changed

Patch-callback tests that assert one callback invocation, the current-update test, and normal unfinished-handler cases use explicit 60s workflow task timeouts. The case that deliberately fails tasks uses 1s, allowing its retry to publish the expected warning failure within the existing 20s assertion. Its background result task is cancelled and awaited at teardown.

Each termination case gets its own task queue. The Java time-skipping server handles poll cancellation asynchronously, allowing a previous worker's poll on a reused queue to claim the next initial task before its intended worker starts. That task holds an internal time-skipping lock until its wall-clock deadline. An isolated queue preserves the required start/signal/update/cancel batching. See the test server's polling implementation.

SDK Core advances from 472f94ee to merged 9cb90f5a, including sdk-rust #1620. Core delivers queued local-activity cancellations before evicting their run, fixing the shutdown hang exposed by the existing cancellation/eviction test. The three included Core commits' release notes are imported, including Nexus replay checks that retain compatibility with older histories.

Testing

SDK 1.34.0, freshly built Core 9cb90f5a, pinned Server 1.32.0-162.0, and Java test server 1.40.0:

• Python 3.14.3 full SDK suite: 1,476 passed locally; 1,124 passed with time skipping.
• Exact CI Python 3.10.11 full suites: 1,465 passed locally; 1,113 passed with time skipping after queue isolation. The prior plain time-skipping run timed out in a termination case and cascaded into 38 failures.
• Controlled prior poll: on the shared queue it takes the initial task and the intended worker gets no activation until the held token is released. With an isolated queue, identical assertions complete in about 0.29s with the same 60s deadline. This models the server cancellation window; the first natural stall's history was unavailable.
• Before the retry helper fix, both named unfinished-handler tests failed with time skipping. Afterward, all 50 related cases passed in each server mode.
• Normal unfinished-handler cases tolerate a 25s initialization stall.
• Identical 200ms replayed-activity delivery delay: old Core completes the workflow but hangs at shutdown with an uncancelled activity; new Core delivers both cancellations, shuts down, and leaves zero activities. This passes on Python 3.10 and 3.14.
• Full poe lint, poe bridge-lint, and changelog fragment validation pass.

All 15 GitHub CI jobs passed on 6d7c33ab. The protobuf job's first attempt could not extract the dev-server archive during fixture setup; a single retry passed on the same commit.

[Copilot]

🟡 Changes recommended

The updated assertion will fail in the Cloud envconfig suite where the local dynamic configuration is not applied.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Raises the local dev server’s default workflow task timeout to reduce CI failures caused by worker stalls.

Changes:

• Configures a 60-second default workflow task timeout.
• Updates workflow-info test expectations.

File summaries

File	Description
tests/conftest.py	Adds the local dev-server timeout configuration.
tests/worker/test_workflow.py	Updates the expected default timeout.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[DABH]

The timeout mechanism still reproduces on current main 76875b8, but I recommend reworking the suite-wide default before merging.

With Python 3.14.3, freshly built matching Core 472f94e, and pinned Server 1.32.0-162.0:

Controlled case, same delay before/after	Current main	With this patch
12s delay before first workflow instance creation	Patch callback runs twice; assertion fails after a WFT timeout	Passes on attempt 1 without a WFT timeout
25s synchronous instance-creation stall, before the activation timer	current_update fails with FAILED_PRECONDITION after WFT retries	Passes without a WFT timeout

Seven targeted controls passed both before and after. These experiments demonstrate tolerance of injected delays, not the frequency of naturally occurring CI stalls. An asynchronous 25s handoff delay did not reproduce the update failure; blocking the Python event loop did.

The inline note documents a measured side effect: an unchanged current-main local-activity cancellation/eviction test passes but slows from 8.42s to 48.47s. Giving that workflow an explicit 5s task timeout restores 4.28s while keeping the server default at 60s.

Drift matters here: #1847's GC changes and #1848's readiness changes have landed since the historical evidence. The latest inspected main run (38319291130) hangs in cancellation tests; its macOS failure follows an intentional task failure/eviction, rather than establishing the historical first-task expiry mechanism. This patch has not been verified as a fix for those current CI failures.

I would first scope longer task timeouts to tests that need stall tolerance. If retaining the global default, give local-activity signal/cancellation tests explicit short task timeouts and reassess the remaining suite's timing assumptions. No new functional test failure was demonstrated in the patched sample.

[DABH]

A fresh queue makes the setup below belong to this case alone. The Java test server processes poll cancellation asynchronously, so a prior poll on a reused queue can take the initial task before the intended worker starts.

In the controlled reproduction, the shared queue leaves that worker with zero activations until the held task is released; the isolated case completes in about 0.29s with the same 60s deadline. The plain Python 3.10.11 time-skipping suite passes all 1,113 tests after this change.

[DABH]

Updated to 6d7c33ab, still one commit. All 15 CI jobs now pass on this revision.

The fixes cover deliberate warning-test retry timing, delivery of local-activity cancellations before eviction through merged Core 9cb90f5a, and isolated queues for termination cases. The queue change preserves their expected activation and warning assertions; its controlled reproduction and full-suite results are documented above and inline.

Full SDK tests on Python 3.10 and 3.14, lint, cloud, FIPS, and package checks passed. The protobuf job needed one retry after server-archive extraction failed during setup; that retry passed on the same commit.