test: add httpx-based Code Interpreter test suite ported from the SDK by devin-ai-integration[bot] · Pull Request #345 · e2b-dev/code-interpreter

devin-ai-integration · 2026-10-07T12:06:19Z

Summary

Ports the Code Interpreter test suite from e2b-dev/E2B (packages/code-interpreter-python / -js) into this repo as a standalone tests/ uv project. The e2b SDK is used only for sandbox lifecycle (Sandbox.create/kill/connect, get_host(49999), access tokens, commands.run for in-sandbox shell checks); everything that talks to the Code Interpreter server goes over its HTTP API on port 49999 with httpx, so the server protocol is what's under test and template/server changes can be validated here without an SDK release. No e2b_code_interpreter import anywhere.

Harness (tests/harness/):

CodeInterpreter / AsyncCodeInterpreter — wire-protocol client for https://{sandbox.get_host(49999)}: /health, /contexts*, and /execute as an NDJSON stream aggregated into Execution (stdout, stderr, results, error, execution_count, completed, raw events). Sends X-Access-Token / E2B-Traffic-Access-Token from the sandbox; non-2xx raises CodeInterpreterError(status_code, body). wait_for_kernel(language) keeps the 4-attempt-on-500 readiness retry for the lazily started Java/R contexts (called inline by the tests that need it).
Fixtures (conftest.py): sandbox / sandbox_factory(timeout=, envs=, allow_public_traffic=) wrap e2b.Sandbox; client / client_factory / async_client build the httpx client for a fresh sandbox (killed on teardown). E2B_DEBUG=true targets http://localhost:49999 (make start-template-server) and skips skip_debug tests.

Coverage ported: basic/statefulness/stdout/stderr/error, !bash magic, execution count, secure-traffic sandboxes, reconnect (Sandbox.connect), contexts (create/list/restart/remove, cwd, alias normalization), kernels (js/ts/r/java/bash, ESM imports, top-level await), cwd per language, env vars (sandbox-level, per-execution, overwrite × python/js/bash/r/java), streaming event order, display data / pandas data / custom repr / HTML / images, all chart types (asserting on the raw chart JSON), client-timeout → kernel interrupt, sandbox killed mid-execution, systemd recovery (commands.run("kill -9 …", user="root"), then poll /health), plus an async smoke subset. New HTTP-level checks the SDK suites couldn't express: /execute → 400 on context_id+language, 404 on unknown context (also for restart/remove).

Jupyter Unix-socket isolation (#343) is tested in a real sandbox instead of in-process on the runner: test_jupyter_socket.py asserts /run/e2b-jupyter is 0700 root and server.sock 0600 root, /api/status answers over the socket as root, user cannot connect (curl exit 7), and nothing listens on tcp 8888 (ss -ltn + connect_ex from the kernel). Execution/interrupt over the socket is covered by the basic/interrupt tests, so server_tests.yml, template/tests/, template/pytest.ini and template/requirements-test.txt are removed.

Deliberately not ported (SDK-internal behavior): sandbox URL resolution, timeout construction, exception formatting, NDJSON line-splitting utils, Execution.to_json(), chart model deserialization. The suite is written once (sync) rather than duplicated sync/async, since the server is the unit under test. The matching upstream trim is e2b-dev/E2B#1956.

CI: new code_interpreter_tests.yml (reusable, run_recovery_tests switch like the SDK workflow) wired into pull_request.yml between build-template and cleanup-build-template; lint.yml syncs tests/ so pnpm run lint/format covers it (added to pnpm-workspace.yaml).

Verified locally: 86 tests pass against code-interpreter-v1 (uv run pytest -n 4, ~40s).

Link to Devin session: https://app.devin.ai/sessions/6fd302f062cc459796016eba4e06d9e1
Open in Devin Desktop: https://app.devin.ai/desktop/session/6fd302f062cc459796016eba4e06d9e1?variant=devin
Requested by: @mishushakov

cla-bot · 2026-10-07T12:06:23Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Devin.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

devin-ai-integration · 2026-10-07T12:06:24Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

chatgpt-codex-connector · 2026-10-07T13:32:30Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T13:35:58.264261Z	`8932b92`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

…r protocol Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

…R tests Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8932b92a1d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

mishushakov · 2026-10-07T13:37:26Z

@claude review

@mishushakov

…dev/code-interpreter (#1956) ## Summary Follow-up to e2b-dev/code-interpreter#345, which ports the Code Interpreter behavior suite into the template repo as a direct-HTTP (httpx) suite. The SDK suites in `packages/code-interpreter-python` and `packages/code-interpreter-js` now only test SDK surface; everything that asserted on server/template behavior is removed here (~-2900 LOC, no `src/` changes). Removed (py sync + async, js): `charts/*` live chart-shape tests (the Python `charts/test_json.py` `to_json` test stays), `test_default_kernels`/`defaultKernels` (js/ts/r/java/bash hello-worlds), `test_cwd`, `test_bash`, `env_vars/*` (per-language × sandbox-level/per-execution/overwrite), `test_statefulness`, `test_execution_count`, `test_display_data`, `test_data`, `test_custom_repr_object`, `images/*`, `test_systemd`, and `test_kernels` (its one SDK check, `test_pass_context_and_language` → `InvalidArgumentException` / `InvalidArgumentError`, moved into `test_contexts` / `contexts.test.ts`). With the java/r kernel tests gone, the `_wait_for_kernel*` / `waitForKernel` readiness-retry helpers and their unit tests (`test_java_kernel_readiness.py`, `javaKernelReadiness.test.ts`) have no callers and are removed from `conftest.py` / `setup.ts` too. Replaced with SDK-focused coverage (from review): - `tests/{sync,async}/test_env_vars.py`, `tests/envVars.test.ts` — one per-execution `envs` → `env_vars` pass-through test per implementation (the HTTP suite can't see the SDK's request mapping). - `tests/charts/test_deserialize.py` — fixture-based unit tests for `_deserialize_chart` / `Result(chart=…)` / `to_json` covering line, scatter, bar, pie, box-and-whisker, nested superchart, unknown type and unknown scale. No JS counterpart: `Result` in JS stores the raw `chart` payload, there is no deserialization step to test. Kept: basic (incl. secure-traffic), callbacks, contexts, interrupt, killed, reconnect, streaming, messaging/utils/sandbox-url/timeout/exception-format unit tests, `performance.py`. CI: `code_interpreter_{python,js}_tests.yml` `run_recovery_tests=false` paths no longer ignore the deleted systemd files (interrupt is the only remaining recovery suite). Verified locally against `code-interpreter-v1`: Python 62 → now 77 incl. the new unit tests, JS 38 passed. Linear: [SDK-386](https://linear.app/e2b/issue/SDK-386/port-code-interpreter-test-suite-from-the-sdk-to-e2b-devcode) Link to Devin session: https://app.devin.ai/sessions/6fd302f062cc459796016eba4e06d9e1 Open in Devin Desktop: https://app.devin.ai/desktop/session/6fd302f062cc459796016eba4e06d9e1?variant=devin Requested by: @mishushakov --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mish@e2b.dev <mish@e2b.dev>

…rver_tests.yml Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

claude

Findings marked 🟡 are optional suggestions and need no follow-up push.

mishushakov · 2026-10-07T13:47:09Z

check the findings above

…local server; fail fast if the kill in test_killed errors Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

linear-code · 2026-10-08T07:09:20Z

SDK-386

devin-ai-integration Bot assigned mishushakov Oct 7, 2026

devin-ai-integration Bot force-pushed the devin/1791374382-httpx-test-suite branch from f8e2abe to f60a587 Compare October 7, 2026 12:07

cla-bot Bot added the cla-signed label Oct 7, 2026

devin-ai-integration Bot mentioned this pull request Oct 7, 2026

test: drop Code Interpreter server-behavior tests now covered in e2b-dev/code-interpreter e2b-dev/E2B#1956

Merged

mishushakov reviewed Oct 7, 2026

View reviewed changes

Comment thread tests/conftest.py Outdated

mishushakov marked this pull request as ready for review October 7, 2026 13:32

devin-ai-integration Bot and others added 4 commits October 7, 2026 13:35

test: add httpx-based Code Interpreter test suite ported from the SDK

93d695a

Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

test: use the e2b SDK for sandbox lifecycle, keep httpx for the serve…

2fed149

…r protocol Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

test: drop java_client fixture, call wait_for_kernel inline like the …

638489d

…R tests Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

test: drop java_client fixture, call wait_for_kernel inline like the …

042d1e8

…R tests Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

devin-ai-integration Bot force-pushed the devin/1791374382-httpx-test-suite branch from 8932b92 to 042d1e8 Compare October 7, 2026 13:35

chatgpt-codex-connector Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread tests/conftest.py

test: move Jupyter Unix socket checks into the sandbox suite, drop se…

51873f4

…rver_tests.yml Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

claude Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread tests/conftest.py

Comment thread tests/test_killed.py Outdated

test: skip sandbox provisioning lazily so E2B_DEBUG mode reaches the …

cab9d70

…local server; fail fast if the kill in test_killed errors Co-Authored-By: mish@e2b.dev <mish@e2b.dev>

mishushakov approved these changes Oct 7, 2026

View reviewed changes

mishushakov merged commit 2dd4042 into main Oct 7, 2026
11 checks passed

mishushakov deleted the devin/1791374382-httpx-test-suite branch October 7, 2026 14:03

Navigation Menu

Repository navigation

test: add httpx-based Code Interpreter test suite ported from the SDK - #345

test: add httpx-based Code Interpreter test suite ported from the SDK#345
mishushakov merged 6 commits into
mainfrom
devin/1791374382-httpx-test-suite

devin-ai-integration Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

cla-bot Bot commented Oct 7, 2026

Uh oh!

devin-ai-integration Bot commented Oct 7, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 7, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

mishushakov commented Oct 7, 2026

Uh oh!

claude Bot left a comment

Uh oh!

Uh oh!

Uh oh!

mishushakov commented Oct 7, 2026

Uh oh!

Uh oh!

linear-code Bot commented Oct 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

devin-ai-integration Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

cla-bot Bot commented Oct 7, 2026

Uh oh!

devin-ai-integration Bot commented Oct 7, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

mishushakov commented Oct 7, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mishushakov commented Oct 7, 2026

Uh oh!

Uh oh!

linear-code Bot commented Oct 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented Oct 7, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 7, 2026 •

edited

Loading