Skip to content

Run task callbacks in fresh interpreters - #73380

Open
keith991001 wants to merge 2 commits into
apache:mainfrom
keith991001:codex/callback-subprocess-exec
Open

keith991001 wants to merge 2 commits into
apache:mainfrom
keith991001:codex/callback-subprocess-exec

Conversation

@keith991001

Copy link
Copy Markdown
Contributor

Reviewed by @keith991001: this change addresses the callback-specific
fork+exec gap described in #73143.

Task callbacks used a closure as their subprocess target, so they could not
opt into WatchedSubprocess's fork+exec path and remained exposed to locks
inherited from supervisor threads.

This change:

  • adds an importable callback child entry point and sends startup data over
    the existing supervisor socket;
  • applies execute_tasks_new_python_interpreter to callback subprocesses;
  • adds a real fork+exec callback regression test and focused unit coverage;
  • updates the configuration documentation and the existing significant
    newsfragment.

Tests:

  • uv run --no-sync --project task-sdk pytest task-sdk/tests/task_sdk/execution_time/test_callback_supervisor.py -q
  • uv run --no-sync prek run mypy-task-sdk --all-files
  • Ruff formatting and lint checks
  • Applicable pre-commit and manual hooks

closes: #73143


Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines


Drafted-by: Codex (GPT-5); reviewed by @keith991001 before posting

Callback processes could inherit locks held by supervisor threads because
their closure-based target forced a bare fork. Giving callbacks an importable
child entry point lets execute_tasks_new_python_interpreter provide them the
same isolation as task processes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Task SDK: importable entry point for callback subprocesses so they can use fork+exec

1 participant