Run the worker in a subprocess so remote bugs cost seconds, not an hour

[rajeeja]

Overview

Every remote defect in the CONUS-RRM work took about half an hour to see: SSH with MFA, a redeploy, a Slurm queue, then one tool call. Four bugs hid behind that latency — a stale checkout, an orphaned PID file, a client timeout shorter than the job, and an OOM-killed worker — and each was only distinguishable from the others after paying the cost again. None of them needed a cluster. They needed a worker that is a separate process with its own memory limit and its own installed packages, which is all a subprocess is.

tests/fake_endpoint.py patches globus_compute_sdk.Executor and runs the submitted function in a child interpreter. It is faithful where it matters and documents where it is not: the function really is serialized and shipped by value, a memory cap really is RLIMIT_AS, and the worker can be pointed at a different PYTHONPATH to reproduce version drift. There is no Slurm, no network and no MPI launcher, so it does not replace test_remote_remap_live.py.

Also fixes what it was built to find

A killed worker normalized to Remote execution failed on chrysalis: *****, because the code kept the last line of the Globus message — which is boilerplate about Python versions, not the cause. Diagnosing the real failure meant bypassing this server entirely. It now names the host, the worker id, and memory as the usual cause. Every _worker_runtime envelope also carries peak_rss_gib, the number today's diagnosis needed and had to get out-of-band.

Two findings from building it

dill ships closures, so my first "closures fail remotely" test was simply wrong; it now pins the opposite, and the real hazard — importing something the worker lacks — is the one asserted. And macOS refuses RLIMIT_AS, RLIMIT_DATA and RLIMIT_RSS, so a cap there would silently not apply; those tests skip on darwin rather than pretend, and CI is Linux.

Verification

• 1287 pass locally, pre-commit clean.
• The normalizer test was checked by sabotage. It initially passed with the fix reverted, because my synthetic message was 268 characters and the normalizer only rewrites messages over 600 — a tidy fixture took a different code path. The fixture is now 6.3k characters, matching the real thing, and the sabotage fails as it should.