Repository navigation
GeneratorExit and asyncio.TaskGroup - missing from _is_base_error? #135736
Description
Bug report
Bug description:
Consider the following code:
import asyncio
async def TestGen():
yield 1
async def TestFn():
async with asyncio.TaskGroup() as tg:
async for n in TestGen():
yield n
async def Run():
g = TestFn()
await g.asend( None )
await g.aclose()
asyncio.run( Run() )which unexpectedly yields the following output:
+ Exception Group Traceback (most recent call last):
| File "<python-input-7>", line 16, in <module>
| asyncio.run( Run() )
| ~~~~~~~~~~~^^^^^^^^^
| File "/usr/lib/python3.13/asyncio/runners.py", line 195, in run
| return runner.run(main)
| ~~~~~~~~~~^^^^^^
| File "/usr/lib/python3.13/asyncio/runners.py", line 118, in run
| return self._loop.run_until_complete(task)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
| File "/usr/lib/python3.13/asyncio/base_events.py", line 719, in run_until_complete
| return future.result()
| ~~~~~~~~~~~~~^^
| File "<python-input-7>", line 14, in Run
| await g.aclose()
| File "<python-input-7>", line 7, in TestFn
| async with asyncio.TaskGroup() as tg:
| ~~~~~~~~~~~~~~~~~^^
| File "/usr/lib/python3.13/asyncio/taskgroups.py", line 71, in __aexit__
| return await self._aexit(et, exc)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/usr/lib/python3.13/asyncio/taskgroups.py", line 173, in _aexit
| raise BaseExceptionGroup(
| ...<2 lines>...
| ) from None
| BaseExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "<python-input-7>", line 9, in TestFn
| yield n
| GeneratorExit
+------------------------------------
Expectation: no "unhandled errors in a TaskGroup" error.
Quick look through taskgroups.py reveals that GeneratorExit might be missing from _is_base_error() method:
# Since Python 3.8 Tasks propagate all exceptions correctly,
# except for KeyboardInterrupt and SystemExit which are
# still considered special.
def _is_base_error(self, exc: BaseException) -> bool:
assert isinstance(exc, BaseException)
return isinstance(exc, (SystemExit, KeyboardInterrupt))Adding GeneratorExit to _is_base_error seems to solve issue:
import asyncio
class TaskGroupWithGeneratorExit(asyncio.TaskGroup):
def _is_base_error(self, exc: BaseException) -> bool:
if super()._is_base_error( exc ):
return True
return isinstance(exc, GeneratorExit)
async def TestGen():
yield 1
async def TestFn():
async with TaskGroupWithGeneratorExit() as tg:
async for n in TestGen():
yield n
async def Run():
g = TestFn()
await g.asend( None )
await g.aclose()
asyncio.run( Run() )The above yields no output, as expected.
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Activity
serhiy-storchaka commentedon Jun 22, 2025
I wonder, should not it be defined as
not isinstance(exc, Exception)? SinceSystemExit,KeyboardInterrupt, andGeneratorExitare the only standard exceptions which are not derived fromException, it is almost the same asisinstance(exc, (SystemExit, KeyboardInterrupt, GeneratorExit)). But it would also work with third-party exceptions not derived fromException.cc @gvanrossum, @1st1, @asvetlov.
gvanrossum commentedon Jun 23, 2025
I'm sure @1st1 did this intentionally. I recall talking about it with him. But I don't recall the exact motivation, only that it struck me as odd at the time, though I think maybe the point was to allow all exceptions to be caught, including BaseExceptions, except for KeyboardError and SystemExit (the comment pretty much says so). If you add GeneratorExit to the list, do all the tests still pass? (Are there enough tests for GeneratorExit?) If you make it
not isinstance(exc, Exception)do any tests fail?Sorry I have no better insight. :-(
luctowers commentedon Sep 27, 2025
@serhiy-storchaka @gvanrossum It should be noted that
TaskGroup._is_base_erroris used in two locations.TaskGroup.__aexit__where exception in thewithends up (this is the issue in the example above)TaskGroup._on_task_donewhere the exceptions from the tasks ends upThe current implementation of
_is_base_errormakes sense to me for the second case, a task raising a GeneratorExit should be treated as a normal exception. For the first case though, it creates this unexpected behaviour in the bug report.gvanrossum-ms commentedon Sep 29, 2025
I've pinged Yury privately. Maybe he has an opinion.
1st1 commentedon Sep 29, 2025
Thanks for pinging me!
Let's start with covering the current logic of task groups and why
_is_base_erroreven exists.TaskGroupwraps all exceptions propagated within it in an exception group. But if aSystemExiterror is raised and would be wrapped in a group, it then wouldn't be intercepted byexcept SystemExitor even perform the intended action which is stopping the application ASAP. That's whyKeyboardInterruptandSystemExitare so special - we can't wrap them.The logic is as follows:
KeyboardInterruptorSystemExitwe memorize that inself._base_error, but proceed business as usualTaskGroup.__aexit__, just before propagating all exceptions wrapped in anExceptionGroup, we check ifself._base_error is not Noneand raise that if it is. If not - raise all collected exceptions in a group.Is this ideal? No, but this is how we could ship task groups without breaking previous asyncio behavior. Besides, it's unlikely that you'll need to do much with the remainder of the task errors if a
SystemExitorKeyboardInterruptis happening.asyncio.CancelledErrorisBaseException(link) and we don't want cancellation to break everything likeSystemExit, so no, we can't do that.I think... this might be the only option we have here, but it is problematic in its own way. The problem is that all trapped exceptions in the
TaskGroupwon't propagate (and be lost), onlyGeneratorExitwould.A few thoughts in no particular order:
CancelledErroror run to completion. It's rare that users manually close generators mid-flight.GeneratorExitcaptured in aTaskGroupis a problem, we can't do that.aclose()is called. To mitigate this:loop.call_exception_handler()on all silenced errors, so that they are at least logged.SystemExit,KeyboardInterrupt, andGeneratorExitinherit fromBaseExceptionGroup, and then we could makeAsyncGenerator.aclose()implementation to re-raise the captured "bag" of errors once the generator is closed. This is harder to implement for all obvious reasons but is probably the best option long term.Summary: let's special case
GeneratorExitin_is_base_error, but add code that callsloop.call_exception_handler()on every error before propagatingGeneratorExit. Potentially we should do that when we raiseSystemExitandKeyboardInterrupttoo.achimnol commentedon Oct 12, 2025
Please have a look at my own solution:
GeneratorExittransparently. But there was nothing to do special inTaskScopeabout it.loop.call_exception_handler()or set a custom exception handler callback for the subtasks spawned by the scope.asyncio.TaskScopeand letasyncio.TaskGroupsubclass it #105011 and aiotools, I'm thinking about whether we should provide a way to switch between callbacks andExceptionGroupby user choice.pythongh-135736: Report suppressed sibling errors for all base errors…
pythongh-135736: Report suppressed sibling errors for all base errors…
Merge branch 'main' into pythongh-135736-taskgroup-generatorexit
pythongh-135736: Fix test_taskgroup_20f to use KeyboardInterrupt, not…
1 remaining item
Merge branch 'main' into pythongh-135736-taskgroup-generatorexit
msullivan commentedon Jul 24, 2026
@1st1 and I discussed this a bunch offline when we ran into this in a project we are working on, and decided that the best approach here is:
This ensures that the
aclosecase works while never swallowing exceptions from subtasks.We can also also preserve some of the existing behavior by introducing a new
TaskGroupGeneratorExitexception type that subtypes bothGeneratorExitandBaseExceptionGroup, which would allow any existing code that expects a BaseExceptionGroup to keep working. That could be skipped though if we don't want to have to add a new exception type.pythongh-135736: Fix interaction between TaskGroup and aclose()
pythongh-135736: Fix interaction between TaskGroup and aclose()
Merge branch 'main' into pythongh-135736-taskgroup-generatorexit
gh-135736: Fix interaction between TaskGroup and aclose() (#154649)
[3.15] gh-135736: Fix interaction between TaskGroup and aclose() (GH-…
pythongh-135736: Restrict fix to reporting suppressed sibling errors
pythongh-135736: Fix interaction between TaskGroup and aclose() (pyth…
gh-135736: call exception handler for all BaseExceptions in asyncio.T…