Skip to content

CI: hermetic spawns in in_process tests (#823) - #1452

Open
Mikola Lysenko (mikolalysenko) wants to merge 3 commits into
mainfrom
agent/fix-ci-hermetic-in-process
Open

Mikola Lysenko (mikolalysenko) wants to merge 3 commits into
mainfrom
agent/fix-ci-hermetic-in-process

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Refs #823 (slice 6 of the tracking issue; #823 stays open for the remaining files).

Problem

16 test files spawned the built socket-patch binary with a bare Command::new and their own SOCKET_* sweep. The sweeps differed from file to file. Some removed SOCKET_TELEMETRY_* and SOCKET_NO_CONFIG, some kept them, and none of them forced SOCKET_NO_UPDATE_CHECK. Six files also kept their own copy of the "remove the four apply_env_toggles vars" workaround.

Change (test-only)

  • hosted_memory_{common,engine}, hosted_superseding_pypi, in_process_redirect{,_pdm,_pnpm,_poetry}, in_process_rollback_{hosted,vendored}, in_process_vendor{,_bun_takeover,_npm_v1_takeover}, mode_migration_{bun,cargo,pypi} and remove_rollback_api_overrides now spawn through tests/common/hermetic.rs, and they leave PENDING_RAW_SPAWNS (92 → 76). The tokio call sites wrap the builder with tokio::process::Command::from.
  • The builder already removes every flag-bound SOCKET_* by name, so the per-file toggle lists and remove_rollback_api_overrides' own SOCKET_ENV_VARS copy and its drift guard are gone.
  • hermetic.rs gains FORCED_OPT_OUTS (the three opt-outs, named once). The env_clear harness in hosted_memory_common re-applies it instead of keeping a hand-copied list. hermetic.rs also gains Extra::Pdm (PDM_* + UV_PROJECT_ENVIRONMENT) and Extra::Poetry (POETRY_*), which replace the private sweeps.
  • cache_env::scrub_ambient_bun_tool_env is the bun suites' shared scrub minus SOCKET_*, used for the CLI child in mode_migration_bun.
  • The pdm and poetry scans remove SOCKET_PYPI_JSON_API by name. Their #[serial] tests set_var it for their own mock, and a parallel spawn could otherwise inherit it between the sweep and the spawn.
  • No overlap with CI: hermetic spawns in scan/vendor tests (#823) #1448 (slice 5): different files, and its PENDING_RAW_SPAWNS lines aren't adjacent to these.

CI cost

No workflow change. A lean PR run keeps the same ~28 Linux jobs and the same job-minutes, before and after.

Evidence

  • cargo test -p socket-patch-cli --all-features over the 18 affected targets (the 16 files plus hosted_memory_{parity,rollout}, which share hosted_memory_common) and spawn_env_hygiene: all pass, with 2 exceptions that come from the sandbox, not this change:
    • 3 in_process_redirect chmod-0555 write-failure tests fail only because the sandbox runs as uid 0. Run as nobody (setpriv --reuid=65534), they pass.
    • e2e_bun_lockb (compiled because cache_env changed) has one test that needs live patches-api.socket.dev, which the sandbox proxy blocks.
  • Hostile shell (SOCKET_DRY_RUN=true SOCKET_JSON=true SOCKET_SILENT=true SOCKET_API_TOKEN=hostile SOCKET_API_URL=http://127.0.0.1:1 SOCKET_PROXY_URL=http://127.0.0.1:1 SOCKET_NO_CONFIG=0 SOCKET_ORG_SLUG=hostile SOCKET_ECOSYSTEMS=cargo SOCKET_OFFLINE=true) over all 18 targets: no failures caused by the hostile env. The only failures were the root/sandbox ones above and 4 that also fail as nobody without the hostile env (a cargo/pipenv home artifact); the same 4 pass as root under the hostile env.
  • spawn_env_hygiene 15/15, including no_new_bare_binary_spawns. cargo fmt --all -- --check is clean.

/code-review high

4 findings, all fixed in d87eb06: the SOCKET_PYPI_JSON_API race in the pdm/poetry scans, the hand-copied opt-outs after env_clear, the inline bun scrub, and the private PDM/Poetry sweeps.

Follow-ups for #823: the remaining 76 PENDING_RAW_SPAWNS files (mostly e2e_*, vex_*, vlt_*), minus the 13 that #1448 removes.

🤖 Generated with Claude Code

https://claude.ai/code/session_016h1P5jNLH36RrvXV8HUQp8


Generated by Claude Code

Assisted-by: Claude Code:claude-opus-5-5
Move the binary spawns in 16 in_process, mode_migration and hosted
test files onto tests/common/hermetic.rs and drop them from the
PENDING_RAW_SPAWNS ratchet. A developer's ambient SOCKET_* vars can no
longer change what these suites exercise.

Refs #823

Assisted-by: Claude Code:claude-opus-5-5
Review follow-ups for the in_process hermetic spawns: name the
builder's forced opt-outs once so the env_clear harness re-applies
them, add Extra::Pdm and Extra::Poetry instead of private sweeps,
add cache_env::scrub_ambient_bun_tool_env for the bun suite's CLI
child, and remove SOCKET_PYPI_JSON_API by name in the pdm and poetry
scans so a serial test's mock URL can't leak into a parallel spawn.

Refs #823

Assisted-by: Claude Code:claude-opus-5-5
@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review October 12, 2026 22:33
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

BugBot review


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit d87eb06. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 12, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Ready for review (burn-down agent)

  • Head: d87eb0649a3b2fb2cc9a6e18aabc6426c751a0bc
  • CI: 35/35 green, 50 path-skipped. Cursor Bugbot is neutral. No failures.
  • Bugbot: reviewed this head, no findings. No open review threads.
  • Reviewer: this PR only changes tests. 16 files now spawn the CLI through tests/common/hermetic.rs, which lowers PENDING_RAW_SPAWNS from 92 to 76. Check FORCED_OPT_OUTS and the new Extra::Pdm/Extra::Poetry sweeps, plus the by-name removal of SOCKET_PYPI_JSON_API in the pdm/poetry scans. That removal fixes a race with the #[serial] set_var tests.
  • Slack announcement not sent: the Slack connector in this session has no send tool.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief

What it does. Sixteen test files used to scrub SOCKET_* by hand before spawning the binary. They now use the shared hermetic builder. Their PDM, Poetry and bun scrubs move into shared helpers, and the spawn_env_hygiene allowlist shrinks to match. Only test code changes.

Risk: low. Only the test harness changes, and every child environment is the same as before or stricter. The builder removes every GLOBAL_ARG_ENV_VARS/LOCAL_ARG_ENV_VARS name, including the race-guard vars. Every var a test set itself is still set after the builder runs. No HOME, XDG, USERPROFILE, APPDATA or proxy handling changed.

Look here

  • tests/common/hermetic.rs:79-87: removal by name, which replaces the per-file race-guard lists
  • tests/hosted_memory_common/mod.rs:414-416:`` env_clear, then the forced opt-outs are applied again
  • tests/in_process_redirect_pdm.rs:679-685 and in_process_redirect_poetry.rs:389-398:```` the builder output is converted to tokio::process::Command, and `SOCKET_PYPI_JSON_API` is now removed by name
  • tests/mode_migration_bun.rs:367-369:````` the new bun scrub leaves the forced SOCKET_* opt-outs alone
  • tests/remove_rollback_api_overrides.rs:144-161:`````` the drift-guard test is deleted, since no hand-kept list is left to drift

Verified

  • Compared every removed scrub with the builder and found no lost env setup.
  • No assertion was removed or weakened.
  • CHANGELOG.md is untouched.
  • Passed locally: spawn_env_hygiene, remove_rollback_api_overrides, hosted_superseding_pypi, in_process_vendor_npm_v1_takeover, hosted_memory_engine and mode_migration_pypi. The other 10 changed targets build with no warnings, but I didn't run their tests.
  • CI: 35/35 green, 50 skipped, 1 neutral.
  • Bugbot is clean on d87eb06.

Changes I made: none.

Open questions (non-blocking): the old seed-then-scrub canaries for SOCKET_OFFLINE and SOCKET_PRESERVE_STATE are gone. The builder still removes both by name, but HOSTILE_SEEDS doesn't seed them, so a regression there would no longer turn the suite red. Seeding them could be a follow-up.

Auto-merge is armed, so approving sends this straight to the merge queue.


Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants