fix(windows): the acceptance harness read UTF-8 JSON as cp1252 - #347
Merged
Merged
Conversation
`Install packages (win32-x64)` has never passed. With these three fixes it
passes end to end on real Windows hardware, `development_override` included.
## The one that broke it
`run_doctor` did `json.loads(out.read_text())`. `Path.read_text()` with no
encoding uses `locale.getpreferredencoding()`, which on Windows is the ANSI
code page -- cp1252 here, not UTF-8.
That file is written by a Rust process, and it carries the INSTALL PATH. The
harness installs under a directory containing a non-ASCII character on purpose
(`tempfile.TemporaryDirectory(prefix='BioRouter installed ü ')`), so the `ü`
came back as `ü` -- its own UTF-8 bytes reinterpreted one at a time. Measured
on Windows Server 2025, from the failure itself:
expected segment: 'BioRouter installed ü tddzmzb0' codepoints [... 0xFC ...]
reported segment: 'BioRouter installed ü tddzmzb0' codepoints [... 0xC3, 0xBC ...]
So the reported executable could never equal the expected one, and the job
failed with "Doctor resolved outside the installed payload" -- blaming the
product for the harness's decoding. Confirmed against the real artefact rather
than inferred: `doctor-cold.json` contains the bytes `C3 BC` and no `\u`
escapes, because serde_json emits raw UTF-8 where Python's `json.dumps`
defaults to `ensure_ascii=True`.
JSON is UTF-8 by RFC 8259, so this was wrong on every platform and merely
invisible where the locale is already UTF-8 -- which is why it survived CI on
Linux and macOS. All thirteen text reads and writes across the three
acceptance scripts now name their encoding.
## Two diagnostics that could not diagnose
Both are why this needed a Windows box instead of being readable from the logs.
`process_tree()` was evaluated inside the error message, i.e. AFTER
`stop_tree(child)` had already killed the process and its descendants. The one
line that exists to say WHAT was hung printed "(no Biorouter-related process
alive) [0 relevant of 136 total processes]" every time. It is captured before
the kill now.
The phase breadcrumbs were written into the caller's `TemporaryDirectory`,
deleted on scope exit. `doctor-cold.err` -- the trace for the attempt that
SUCCEEDED, which is the one that localises where the time went -- was
discarded before anything could upload it. Downloading
`installed-packages-win32-x64` from run 35474721330 yields `backends.json` and
nothing else. They are written to `report_path.parent` now, which is
`target/package-acceptance/`, the directory the workflow already uploads. That
directory held 1 file before this change and 50+ after.
## A Windows bug in the build tooling
`scripts/computer-use-runtime.py` called bare `shutil.rmtree` at three sites,
and it CANNOT delete a git checkout on Windows: git marks pack files read-only
and Windows refuses to unlink a read-only file.
PermissionError: [WinError 5] Access is denied:
'...\source.noindex\.git\objects\pack\pack-9a1245af....idx'
Invisible on Linux and macOS, where deletion is governed by the directory's
write bit. The effect is that `computer-use-runtime.py build` worked exactly
ONCE on a Windows machine and failed on every later run -- while cleaning up
the previous one, which reads as a corrupted checkout rather than a permission
bit. All three sites go through `remove_tree`, which clears the bit and
retries, and re-raises anything that still will not go.
## Verified
Full `computer-use-package-acceptance.py build/verify win32-x64` at commit
89eca09, with the genuine `package-backends-x86_64-pc-windows-gnu` artefact
from run 35474721330 so HEAD matched the provenance stamp -- the same job CI
runs, on hardware:
VERIFY_EXIT=0, zero tracebacks
doctor status ready, development_override False, integrity verified
permissions accessibility True, screen_recording True
cold 4.83s exit 0 (budget 40s), warm 2.02s exit 0 (budget 20s)
10 tools
Before the encoding fix the same command failed with the payload-path error.
Unit suite 20 -> 27 tests, all passing, with controls both ways: reinstating
`out.read_text()` makes the census name that exact line, and a bare
`shutil.rmtree` over a read-only tree raises `PermissionError` on Windows while
`remove_tree` succeeds.
⚠ Two things this does NOT fix, stated so nobody reads more into a green run.
The 34.91s probe timeout seen on the hosted runner does not reproduce on real
hardware in ANY condition I could construct -- interactive, session 0 with no
desktop, the space-and-non-ASCII install path, or with Defender fully enabled
(0.89s). It remains runner-specific and unexplained, and the harness fixes
above are what will localise it next time. And `--expect-status` is deliberately
untouched: a hosted runner reaches neither `ready` nor `desktop_unavailable`, it
times out, so relaxing the expected status would not make the job meaningful.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Install packages (win32-x64)has never passed. With these three fixes it passes end to end on real Windows hardware,development_overrideincluded.The one that broke it
run_doctordidjson.loads(out.read_text()).Path.read_text()with no encoding useslocale.getpreferredencoding()— on Windows the ANSI code page, cp1252, not UTF-8.That file is written by a Rust process and carries the install path. The harness installs under a directory with a non-ASCII character on purpose (
tempfile.TemporaryDirectory(prefix='BioRouter installed ü ')), so theücame back asü— its own UTF-8 bytes reinterpreted one at a time. Measured on Windows Server 2025, from the failure itself:The reported executable could therefore never equal the expected one, and the job failed with "Doctor resolved outside the installed payload" — blaming the product for the harness's decoding.
Confirmed against the real artefact rather than inferred:
doctor-cold.jsoncontains the bytesC3 BCand no\uescapes, because serde_json emits raw UTF-8 where Python'sjson.dumpsdefaults toensure_ascii=True. That difference is why my first regression test passed for the wrong reason until the control caught it.JSON is UTF-8 by RFC 8259, so this was wrong on every platform and merely invisible where the locale is already UTF-8 — which is why it survived CI on Linux and macOS. All thirteen text reads and writes across the three acceptance scripts now name their encoding.
Two diagnostics that could not diagnose
Both are why this needed a Windows box rather than being readable from the logs.
process_tree()ran afterstop_tree(child). It was evaluated inside the error message, by which point the process and its descendants were already killed. The one line that exists to say what was hung printed(no Biorouter-related process alive) [0 relevant of 136 total processes]every time. Captured before the kill now.TemporaryDirectoryand deleted.doctor-cold.err— the trace for the attempt that succeeded, which is the one that localises where the time went — was discarded before anything could upload it. Downloadinginstalled-packages-win32-x64from run 35474721330 yieldsbackends.jsonand nothing else. They go toreport_path.parentnow (target/package-acceptance/, already uploaded): that directory held 1 file before this change and 50+ after.A Windows bug in the build tooling
scripts/computer-use-runtime.pycalled bareshutil.rmtreeat three sites, and it cannot delete a git checkout on Windows — git marks pack files read-only and Windows refuses to unlink a read-only file:Invisible on Linux/macOS, where deletion is governed by the directory's write bit. Effect:
computer-use-runtime.py buildworked exactly once on a Windows machine and failed on every later run — while cleaning up the previous one, which reads as a corrupted checkout rather than a permission bit.Verified
Full
computer-use-package-acceptance.py build/verify win32-x64at commit89eca096, with the genuinepackage-backends-x86_64-pc-windows-gnuartefact from run 35474721330 soHEADmatched the provenance stamp — the same job CI runs, on hardware:Before the encoding fix, the same command failed with the payload-path error.
Unit suite 20 → 27 tests, with controls both ways: reinstating
out.read_text()makes the census name that exact line, and a bareshutil.rmtreeover a read-only tree raisesPermissionErroron Windows whileremove_treesucceeds.What this does NOT fix
⚠ The 34.91s probe timeout on the hosted runner does not reproduce on real hardware in any condition I could construct: interactive, session 0 with no desktop, the space-and-non-ASCII install path, or with Defender fully enabled (0.89s). It stays runner-specific and unexplained — and the harness fixes above are what will localise it next time.
⚠
--expect-statusis deliberately untouched. A hosted runner reaches neitherreadynordesktop_unavailable; it times out. Relaxing the expected status would not make the job meaningful, it would just make it green.