fix(gateway): a container deploy cancelled mid-create leaves no containers running

[earakely-scale]

A gateway deploy in containers that's cancelled while it creates them no longer leaves containers running on Modal.

Why

A chaos run of #169 cancelled a Modal container gateway deploy 6 s in and left two of its containers running until their timeout. Two windows leak:

• The parallel batch. _deploy_via_containers creates the MCP servers and the pgweb and db-mcp sidecars with one asyncio.gather, and added them to _container_sandboxes only once the whole batch returned. A cancel while the batch was pending dropped the containers already made, so the close _run gives a cancelled deploy never saw them. The service-db container, recorded as it's created, was torn down.
• The create request. Modal can create a sandbox after the client has stopped waiting for it. A cancel that lands during Sandbox._experimental_create gave up on the request, and the sandbox Modal made ran until its timeout.

Prod's Modal container gateways run through the same code; a run cancelled at the wrong moment pays for its containers until their timeout.

What

• Each container is recorded as it's created. _provision_mcp and _provision_sidecar add their container to _container_sandboxes the moment create_container returns, as _stand_up_servicedb already did, so a deploy that fails or is cancelled mid-batch closes every container it made.
• A cancelled Modal create cleans up after itself. ModalSandboxProvider.create_container runs the create request to its end even when it's cancelled (asyncio.shield), then terminates what Modal created and re-raises the cancellation; a terminate that fails logs a warning naming the sandbox, which then runs until its timeout. The cancel waits for the request 30 s at most (it takes a few seconds as a rule), so a request that hangs can't hold up a chain's provisioning deadline, a Ctrl-C or an activity's cancel; past that, or on a second cancel, the cancel goes on, a warning says so, and the sandbox the request creates is terminated when it ends, including when it ends in the same step the wait gives up. A process that exits with the request still pending can't clean up after it: a warning names the create, and a sandbox Modal makes for it runs until its timeout. A cancel after the request returned was already handled: that path terminates the sandbox on any BaseException. A second Ctrl-C during the wait stops it, as it stops the rest of a teardown.

Tests

Unit tier and the protocol suite: 7,464 passed. New:

• a container gateway deploy cancelled, through _run, while db-mcp is still being created: the service-db, MCP server and pgweb containers it made are all terminated;
• a Modal create cancelled while the request is in flight: the sandbox Modal returns afterwards is terminated, and it isn't waited on to become ready;
• and when that terminate fails, a warning names the sandbox and the error, and the cancellation is still raised;
• a create whose request hangs, cancelled once (past the wait) or twice: the cancel goes on without the request, and the sandbox it returns afterwards is terminated;
• a create whose request finishes in the same step the wait gives up: its sandbox is terminated;
• a process that exits while a cancelled create's request still hangs: a warning names the create.

Live check

From this machine on Modal, a bundle whose MCP server deploys through the container gateway with local Postgres state, cancelled with Ctrl-C partway through its deploy:

• Before this change: the chaos run's cancel at 6 s left two containers running.
• With only the first half (recording): five cancels between 5 s and 7 s left one sandbox running, from a create cancelled in flight. That's what the second half fixes.
• With both: nine cancels between 5.6 s and 7.2 s landed throughout the create window: before the batch, with one to three of its creates in flight, and with the gateway's own create in flight. Every run exited 130, and a minute later Modal's App had no sandbox running. Again with the 30 s bound: five cancels between 5.8 s and 7.0 s, two with creates in flight, and none left running.

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR still needs the shutdown cleanup gap fixed before merging.

Findings

1.  CLI shutdown drops cleanup ▶

Fix with agent prompt

### Issue 1
src/agent_env/providers/sandbox_providers/modal_sandbox.py:394-397
When a cancelled create exceeds the 30-second wait, cleanup is left to a background callback. The deploy CLI uses `asyncio.run()`, which cancels pending tasks when the deploy exits. If the create is still pending, `_terminate_when_created` skips it because `creating` was cancelled. Modal can still create the sandbox remotely, leaving it running until its timeout.

Keep cleanup alive beyond the deploy’s event loop, or retain enough information to recover and terminate the sandbox after shutdown.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR records gateway containers as they are created and cleans up Modal sandboxes returned after cancellation.

• Gateway deploys record each container as soon as it is created.
• Cancelled Modal creates clean up sandboxes that return late.

_{Reviews (4) · Last reviewed commit: "Terminate a sandbox created just as a ca..." · Reviewed by Greptile}

[greptile-apps]

CLI shutdown drops cleanup

When a cancelled create exceeds the 30-second wait, cleanup is left to a background callback. The deploy CLI uses asyncio.run(), which cancels pending tasks when the deploy exits. If the create is still pending, _terminate_when_created skips it because creating was cancelled. Modal can still create the sandbox remotely, leaving it running until its timeout.

Keep cleanup alive beyond the deploy’s event loop, or retain enough information to recover and terminate the sandbox after shutdown.

Knowledge Base Used: Provider integrations and sandbox deployment

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agent_env/providers/sandbox_providers/modal_sandbox.py
Line: 394-397

Comment:
**CLI shutdown drops cleanup**

When a cancelled create exceeds the 30-second wait, cleanup is left to a background callback. The deploy CLI uses `asyncio.run()`, which cancels pending tasks when the deploy exits. If the create is still pending, `_terminate_when_created` skips it because `creating` was cancelled. Modal can still create the sandbox remotely, leaving it running until its timeout.

Keep cleanup alive beyond the deploy’s event loop, or retain enough information to recover and terminate the sandbox after shutdown.

**Knowledge Base Used:** [Provider integrations and sandbox deployment](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/agentenv-framework/-/docs/provider-integrations.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[earakely-scale]

Valid, and it is the limit this change accepts. Waiting with no limit is what let a hung request hold up a deadline or a Ctrl-C, so the cap stays. A process that exits cannot run cleanup after it has exited. Modal's sandbox timeout bounds a sandbox that is created after that point. 2adc73f makes such a sandbox traceable: when the callback sees the create was cancelled before it ended (as asyncio.run does at exit), it logs a warning with the create's image, app and region. test_a_process_that_exits_while_a_cancelled_create_hangs_names_the_create_in_a_warning covers this.

[greptile-apps]

Issue 1 is understood and does not require another code change. The 30-second bound intentionally allows cancellation to proceed when Modal does not answer; if the process then exits, no in-process cleanup can run, and the sandbox’s configured Modal timeout is the remaining safety bound. Commit 2adc73f makes that residual case observable by warning when the create was cancelled before Modal answered, including the image/app/region context needed to locate the sandbox.

Issue 2 was valid and is fixed in 2adc73f: the late-termination callback is now registered even when the create has completed by the time the bounded wait ends. Since add_done_callback runs immediately for an already-completed future, a sandbox returned in that race is terminated rather than leaked. The added regression test covers this case.