Skip to content

fix(windows): the acceptance harness read UTF-8 JSON as cp1252 - #347

Merged
Broccolito merged 1 commit into
mainfrom
fix/windows-acceptance-encoding
Sep 20, 2026
Merged

Broccolito merged 1 commit into
mainfrom
fix/windows-acceptance-encoding

Conversation

@Broccolito

Copy link
Copy Markdown
Collaborator

Install packages (win32-x64) has never passed. With these three fixes it passes end to end on real Windows hardware, development_override included.

The one that broke it

run_doctor did json.loads(out.read_text()). Path.read_text() with no encoding uses locale.getpreferredencoding() — on Windows the ANSI code page, cp1252, not UTF-8.

That file is written by a Rust process and carries the install path. The harness installs under a directory with a non-ASCII character on purpose (tempfile.TemporaryDirectory(prefix='BioRouter installed ü ')), so the ü came back as ü — its own UTF-8 bytes reinterpreted one at a time. Measured on Windows Server 2025, from the failure itself:

expected segment: 'BioRouter installed ü tddzmzb0'   codepoints [... 0xFC ...]
reported segment: 'BioRouter installed ü tddzmzb0'  codepoints [... 0xC3, 0xBC ...]

The reported executable could therefore never equal the expected one, and the job failed with "Doctor resolved outside the installed payload" — blaming the product for the harness's decoding.

Confirmed against the real artefact rather than inferred: doctor-cold.json contains the bytes C3 BC and no \u escapes, because serde_json emits raw UTF-8 where Python's json.dumps defaults to ensure_ascii=True. That difference is why my first regression test passed for the wrong reason until the control caught it.

JSON is UTF-8 by RFC 8259, so this was wrong on every platform and merely invisible where the locale is already UTF-8 — which is why it survived CI on Linux and macOS. All thirteen text reads and writes across the three acceptance scripts now name their encoding.

Two diagnostics that could not diagnose

Both are why this needed a Windows box rather than being readable from the logs.

  • process_tree() ran after stop_tree(child). It was evaluated inside the error message, by which point the process and its descendants were already killed. The one line that exists to say what was hung printed (no Biorouter-related process alive) [0 relevant of 136 total processes] every time. Captured before the kill now.
  • The phase breadcrumbs were written into a TemporaryDirectory and deleted. doctor-cold.err — the trace for the attempt that succeeded, which is the one that localises where the time went — was discarded before anything could upload it. Downloading installed-packages-win32-x64 from run 35474721330 yields backends.json and nothing else. They go to report_path.parent now (target/package-acceptance/, already uploaded): that directory held 1 file before this change and 50+ after.

A Windows bug in the build tooling

scripts/computer-use-runtime.py called bare shutil.rmtree at three sites, and it cannot delete a git checkout on Windows — git marks pack files read-only and Windows refuses to unlink a read-only file:

PermissionError: [WinError 5] Access is denied:
  '...\source.noindex\.git\objects\pack\pack-9a1245af....idx'

Invisible on Linux/macOS, where deletion is governed by the directory's write bit. Effect: computer-use-runtime.py build worked exactly once on a Windows machine and failed on every later run — while cleaning up the previous one, which reads as a corrupted checkout rather than a permission bit.

Verified

Full computer-use-package-acceptance.py build/verify win32-x64 at commit 89eca096, with the genuine package-backends-x86_64-pc-windows-gnu artefact from run 35474721330 so HEAD matched the provenance stamp — the same job CI runs, on hardware:

VERIFY_EXIT=0, zero tracebacks
doctor status ready, development_override False, integrity verified
permissions accessibility True, screen_recording True
cold 4.83s exit 0 (budget 40s), warm 2.02s exit 0 (budget 20s)
10 tools

Before the encoding fix, the same command failed with the payload-path error.

Unit suite 20 → 27 tests, with controls both ways: reinstating out.read_text() makes the census name that exact line, and a bare shutil.rmtree over a read-only tree raises PermissionError on Windows while remove_tree succeeds.

What this does NOT fix

⚠ The 34.91s probe timeout on the hosted runner does not reproduce on real hardware in any condition I could construct: interactive, session 0 with no desktop, the space-and-non-ASCII install path, or with Defender fully enabled (0.89s). It stays runner-specific and unexplained — and the harness fixes above are what will localise it next time.

--expect-status is deliberately untouched. A hosted runner reaches neither ready nor desktop_unavailable; it times out. Relaxing the expected status would not make the job meaningful, it would just make it green.

`Install packages (win32-x64)` has never passed. With these three fixes it
passes end to end on real Windows hardware, `development_override` included.

## The one that broke it

`run_doctor` did `json.loads(out.read_text())`. `Path.read_text()` with no
encoding uses `locale.getpreferredencoding()`, which on Windows is the ANSI
code page -- cp1252 here, not UTF-8.

That file is written by a Rust process, and it carries the INSTALL PATH. The
harness installs under a directory containing a non-ASCII character on purpose
(`tempfile.TemporaryDirectory(prefix='BioRouter installed ü ')`), so the `ü`
came back as `ü` -- its own UTF-8 bytes reinterpreted one at a time. Measured
on Windows Server 2025, from the failure itself:

    expected segment: 'BioRouter installed ü tddzmzb0'   codepoints [... 0xFC ...]
    reported segment: 'BioRouter installed ü tddzmzb0'  codepoints [... 0xC3, 0xBC ...]

So the reported executable could never equal the expected one, and the job
failed with "Doctor resolved outside the installed payload" -- blaming the
product for the harness's decoding. Confirmed against the real artefact rather
than inferred: `doctor-cold.json` contains the bytes `C3 BC` and no `\u`
escapes, because serde_json emits raw UTF-8 where Python's `json.dumps`
defaults to `ensure_ascii=True`.

JSON is UTF-8 by RFC 8259, so this was wrong on every platform and merely
invisible where the locale is already UTF-8 -- which is why it survived CI on
Linux and macOS. All thirteen text reads and writes across the three
acceptance scripts now name their encoding.

## Two diagnostics that could not diagnose

Both are why this needed a Windows box instead of being readable from the logs.

`process_tree()` was evaluated inside the error message, i.e. AFTER
`stop_tree(child)` had already killed the process and its descendants. The one
line that exists to say WHAT was hung printed "(no Biorouter-related process
alive) [0 relevant of 136 total processes]" every time. It is captured before
the kill now.

The phase breadcrumbs were written into the caller's `TemporaryDirectory`,
deleted on scope exit. `doctor-cold.err` -- the trace for the attempt that
SUCCEEDED, which is the one that localises where the time went -- was
discarded before anything could upload it. Downloading
`installed-packages-win32-x64` from run 35474721330 yields `backends.json` and
nothing else. They are written to `report_path.parent` now, which is
`target/package-acceptance/`, the directory the workflow already uploads. That
directory held 1 file before this change and 50+ after.

## A Windows bug in the build tooling

`scripts/computer-use-runtime.py` called bare `shutil.rmtree` at three sites,
and it CANNOT delete a git checkout on Windows: git marks pack files read-only
and Windows refuses to unlink a read-only file.

    PermissionError: [WinError 5] Access is denied:
      '...\source.noindex\.git\objects\pack\pack-9a1245af....idx'

Invisible on Linux and macOS, where deletion is governed by the directory's
write bit. The effect is that `computer-use-runtime.py build` worked exactly
ONCE on a Windows machine and failed on every later run -- while cleaning up
the previous one, which reads as a corrupted checkout rather than a permission
bit. All three sites go through `remove_tree`, which clears the bit and
retries, and re-raises anything that still will not go.

## Verified

Full `computer-use-package-acceptance.py build/verify win32-x64` at commit
89eca09, with the genuine `package-backends-x86_64-pc-windows-gnu` artefact
from run 35474721330 so HEAD matched the provenance stamp -- the same job CI
runs, on hardware:

    VERIFY_EXIT=0, zero tracebacks
    doctor status ready, development_override False, integrity verified
    permissions accessibility True, screen_recording True
    cold 4.83s exit 0 (budget 40s), warm 2.02s exit 0 (budget 20s)
    10 tools

Before the encoding fix the same command failed with the payload-path error.

Unit suite 20 -> 27 tests, all passing, with controls both ways: reinstating
`out.read_text()` makes the census name that exact line, and a bare
`shutil.rmtree` over a read-only tree raises `PermissionError` on Windows while
`remove_tree` succeeds.

⚠ Two things this does NOT fix, stated so nobody reads more into a green run.
The 34.91s probe timeout seen on the hosted runner does not reproduce on real
hardware in ANY condition I could construct -- interactive, session 0 with no
desktop, the space-and-non-ASCII install path, or with Defender fully enabled
(0.89s). It remains runner-specific and unexplained, and the harness fixes
above are what will localise it next time. And `--expect-status` is deliberately
untouched: a hosted runner reaches neither `ready` nor `desktop_unavailable`, it
times out, so relaxing the expected status would not make the job meaningful.
@Broccolito Broccolito added the windows Windows-specific defect or platform behaviour label Sep 20, 2026
@Broccolito
Broccolito merged commit 8caee3a into main Sep 20, 2026
22 checks passed
@Broccolito
Broccolito deleted the fix/windows-acceptance-encoding branch September 20, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

windows Windows-specific defect or platform behaviour

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant