Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #40507 +/- ##
=============================================
- Coverage 79.83% 56.60% -23.24%
- Complexity 0 2288 +2288
=============================================
Files 494 1131 +637
Lines 83575 179665 +96090
Branches 0 1487 +1487
=============================================
+ Hits 66722 101696 +34974
- Misses 16853 75448 +58595
- Partials 0 2521 +2521
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
📋 Review Summary
This PR fixes a race condition in AsyncWrapper.teardown() where the wrapped DoFn's resources could be torn down while background thread-pool futures or asyncio tasks owned by that wrapper instance were still running or cleaning up after cancellation. The per-instance tracking of futures and asyncio tasks (while keeping the shared executor and event loop alive for other instances) and the accompanying unit and DirectRunner tests are well-structured.
🔍 General Feedback
- Consider guarding
teardown()againstAsyncWrapper._event_loopbeingNone(e.g., if_sync_fn.setup()fails before the event loop is started, or ifreset_state()is called prior toteardown()) and initializing_pending_futures/_pending_async_tasksin__init__as well assetup().
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
|
Assigning reviewers: R: @damccorm for label python. This pull request likely touches a core component ("core" label). Please review with scrutiny. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
@dustin12 would you mind taking a first look at this one? |
|
R: @dustin12 |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
AsyncWrapper.teardown() can close the resources of the wrapped DoFn while background work still uses them. Track the work that each wrapper owns and wait for it before tearing down the wrapped DoFn. Asyncio tasks stay tracked through cancellation cleanup, and the shared executor and event loop remain available to other wrappers. Without an event loop, no work of this instance can still be running, so teardown does not wait.
ae890e8 to
06ef4d1
Compare
AsyncWrapper.teardown()can close the wrapped DoFn's resources while background work still uses them. A bounded DirectRunner pipeline reportsDONE, then a delayed write fails withValueError: I/O operation on closed file.This change tracks work owned by each wrapper and waits for it before tearing down the wrapped DoFn. Asyncio tasks stay tracked through cancellation cleanup, and the shared executor and event loop remain available to other wrappers.
The AsyncWrapper tests and DirectRunner reproductions passed locally in both thread-pool and asyncio modes. Teardown has no timeout, so hung processing or cleanup can delay shutdown. Pending-output delivery and timer behavior are unchanged.
Run the Python unit suite with:
addresses #123orfixes #456).CHANGES.mdwith noteworthy changes.See CONTRIBUTING.md for contribution guidelines.