search-quality gate is unrunnable on Windows: select() on a pipe, verbatim paths (silent 0.000), autocrlf hash pin, and ORT dll auto-detect

[Qiiks]

The search-quality gate cannot run on Windows: four harness defects, plus a documented platform limitation that fails opaquely

run_search_quality.py is the gate that decides a ranking change (#322). On Windows it produces no number at all. Four of the reasons are ordinary Windows-suite defects with no upstream handling anywhere; the fifth is documented deliberate design that nevertheless fails with an opaque digest instead of telling a Windows reader what it is.

Reproduced on Windows 11, Python 3.14, core.autocrlf=true, at 4c13f9d and again at upstream 7a93615.

---

1. select.select() on a pipe — WinError 10038 (hard stop, first stage)

run.py:154:

ready, _, _ = select.select([self.proc.stdout], [], [], 0.1)

Windows select() accepts sockets only. On the pipe it raises OSError: [WinError 10038], so the first request in run_exact_recall.py dies and the gate exits 2 before any fixture runs:

quality_command: ... run_exact_recall.py --check-corpus ...      # ok
quality_command: ... run_exact_recall.py --binary ... --out ...  # dies
[WinError 10038] An operation was attempted on something that is not a socket
quality_exit:2

run_real_query.py:102 has the same call in its own NdjsonClient. Because run_search_quality.py spawns each stage as python <script>.py, that second client executes as __main__, so patching module attributes of the imported name does not reach it — the primitive has to change, not the caller.

2. normalize_result_path on verbatim paths — silent 0.000

run.py:

def normalize_result_path(raw_path: str, project_root: Path) -> str:
    path = Path(raw_path)
    if path.is_absolute():
        try:
            return path.resolve().relative_to(project_root).as_posix()
        except ValueError:
            return path.as_posix()
    return path.as_posix()

AFT returns Windows paths in \\?\ verbatim form. Path.resolve() preserves the prefix, relative_to() raises, and the fallback returns the raw absolute verbatim string while expected_top_files is relative. Verified against a real repo:

raw : \\?\C:\tmp\aft-src\benchmarks\aft-search\.bench\repos\ripgrep\GUIDE.md
norm: //?/C:/tmp/aft-src/benchmarks/aft-search/.bench/repos/ripgrep/GUIDE.md

Every comparison then misses, and the gate reports all-None on a healthy index — no exception, no warning, well-formed report:

|fastify|sentence|0|2|0.000|0|
Sentence rank-1: **0.000** (baseline 1.000)
quality_exit:1

That is indistinguishable from a catastrophic ranking regression. It is also why run-fusion-quality reported current = 0.000 for me until I caught it (#325 item 2).

3. real-query-vectors.json fails its own hash pin under core.autocrlf=true

run_real_query.py:483 verifies the checked-in pack against the manifest's embedding_pack_sha256. That hash is of LF bytes; a default Windows checkout writes CRLF, so the check fails:

as-checked-out (autocrlf=true) has CRLF : True
hash matches pin as-checked-out          : False
hash matches pin after LF-normalize      : True

corpus_vector_model_mismatch:benchmarks/aft-search/real-query-vectors.json
quality_exit:2

No eol attribute covers it:

$ git check-attr text eol -- benchmarks/aft-search/real-query-vectors.json
... text: unspecified
... eol: unspecified

A .gitattributes entry (text eol=lf) fixes it at source and also protects the sibling_digests check in imports_resolve.py.

4. ensure_ort_env() never looks for onnxruntime.dll

run-fusion-quality:131:

lib_name = "libonnxruntime.dylib" if sys.platform == "darwin" else "libonnxruntime.so"

On Windows the managed runtime is onnxruntime.dll at
%LOCALAPPDATA%\cortexkit\aft\onnxruntime\1.24.4\onnxruntime.dll — present and installed. The helper cannot see it, so the semantic index fails with ONNX-version guidance mentioning v1.17. Setting ORT_DYLIB_PATH by hand works.

This is the only sys.platform check in the entire benchmark. Nothing else is platform-aware.

---

5. The real-query gate is Unix-only in practice — and says so opaquely

Once 1–4 are cleared, exact-recall and concept-recall reach baseline, and the real-query stage dies with:

422 {"error":"vector_missing:query:c83d0361...:aft-search-template-v1|corpus:30d4a64f...:c83d0361...:aft-search-template-v1"}

This is not a defect — it is documented design. crates/aft/src/semantic_index.rs:8143, on the test that would otherwise catch it:

// Unix-only: chunk embed text bakes the OS-native relative path into its
// header (file-summary chunks), so a Windows run hashes "tests\fixtures\…"
// and can never reproduce the unix-captured baseline even with LF-forced
// sources. The property under test — the query-free Rust walk reproduces
// the old RS_QUERY output byte-for-byte — is platform-independent and is
// pinned where the baseline was captured.
#[cfg(unix)]
#[test]
fn rust_semantic_fixture_output_matches_query_baseline() {

The pack and baseline are Unix-captured artifacts, so a Windows run cannot build the index. I confirmed it is not revision drift (same result at 4c13f9d and upstream 7a93615), and CI runs the gate on ubuntu-latest, which is why the gap is invisible there.

But upstream's comment understates the scope

Upstream says "file-summary chunks". Measured by logging every miss rather than stopping at the first:

misses enumerated                    : 27171
  of which log-truncated (excluded)  : 17898
  fully captured                      : 9273
    backslash form resolves in pack  :     0
    POSIX-slash form resolves in pack:  8997
    neither resolves                 :   276

The first four misses from the server's own log:

file:tests\windows-e2e\mock-server.js kind:file-summary ...           <- file-summary
name:fs file:tests\windows-e2e\mock-server.js kind:variable ...       <- variable
name:path file:tests\windows-e2e\mock-server.js kind:variable ...     <- variable
name:parsePort file:tests\windows-e2e\mock-server.js kind:function ...<- function

That matches the source: build_embed_text_with_lines_and_caps bakes relative into the code-chunk header too (semantic_index.rs:7303 upstream), using the same strip_prefix(project_root).to_string_lossy() as the file-summary path at :7521. The build only stops on a file-summary chunk because that is the first chunk of the first nested file — a file-summary-only fix would die on the very next code chunk. Whoever addresses this should normalise at the shared relative-path computation.

Residual: 276 fully-captured misses the separator does not explain. They are not a key-format problem — every one has a file: that exists in the evidence tree, and they concentrate in 169 files (18 in bash_rewrite_diff/corpus.toml, 6 in aft-cli/src/lib/sanitize.ts). Trailing-newline and CRLF-normalised variants do not resolve either. That pattern reads as a per-file content divergence, separate from the separator question, and is where I would look next.

The actionable part of item 5

A one-line platform assertion in run_real_query.py before the index build would turn a half-hour archaeology session into a sentence: "the real-query reference pair is Unix-captured and cannot be evaluated on this platform."

---

What I did about it

A sitecustomize.py on PYTHONPATH patching the two primitives rather than the callers — pipe-aware select.select via PeekNamedPipe, and normalize_result_path stripping the verbatim prefix — so both client classes and any future one are covered without touching the bench. With that, plus ORT_DYLIB_PATH and an LF-normalised pack:

|Repository|Family|Passed|Total|Recall|Exact markers|
|fastify   |sentence|2|2|1.000|2|
... all four repos, both families, 1.000
Sentence rank-1: 1.000 (baseline 1.000)
Pair recall@10 : 1.000 (baseline 1.000)
quality_exit:0

So the harness is sound on Windows once these are cleared — items 1–4 are the whole gap for the stages that can run.

Method note: the miss enumeration logged through the fixture server and continued on synthetic vectors purely so the full affected set could be seen. That run must never be scored — substituting a synthetic vector for a pinned one makes ranking meaningless, which is exactly why the gate refuses a miss instead of inventing one. I stopped it and reverted embedding_fixture_server.py; no number from it is cited above. Related harness defects: #325.

Also worth recording: driving these scripts from an external runner, patches must be installed before importing the harness, because it does from run import normalize_result_path and binds the name at import time. Patching run.normalize_result_path afterwards leaves the harness holding the original — the same silent 0.000.

[aft-alfonso]

All five confirmed from the source, and your split between the four defects and the one design decision is the right one. Taking them in your order.

1. select() on a pipe: the primitive changes, not the callers, for the reason you give — the second client runs as __main__ and a module patch never reaches it. The NDJSON client gets a reader thread feeding a queue, which is the portable shape; select goes away entirely rather than gaining a Windows branch.

2. Verbatim paths: fixed at the point where the result path enters the harness, shared with aft-search bench: the harness indexes its own answer key; normalize_result_path breaks on Windows verbatim paths #325, and the harness will refuse to write an all-None report on a non-empty index rather than emit the silent 0.000.

3. The vector pack hash: .gitattributes gets benchmarks/aft-search/*.json text eol=lf (and the digest-checked siblings), so the checked-out bytes are the pinned bytes on every autocrlf setting. That is the same class as the CRLF fixture pins already in the file.

4. ensure_ort_env(): it will resolve onnxruntime.dll under %LOCALAPPDATA%\cortexkit\aft\onnxruntime\<version>\ on Windows, and the three-way platform switch becomes the only platform-aware code in the bench, in one place.

5. On the real-query gate: your measurement is right and the upstream comment was understating it — the relative path is baked into every chunk header, code chunks included, at the shared strip_prefix(project_root) site, so a file-summary-only change would fail on the next chunk. I am not re-deciding the separator on this issue, but two things follow from what you found. First, the platform assertion you ask for lands: run_real_query.py will say, before any index build, that the reference pair is Unix-captured and cannot be evaluated on this platform, and exit with a distinct code. Second, the fix that makes the pair platform-independent is to normalize the path to POSIX separators at that one shared site before it enters the embed text — every existing Unix-captured vector hashes byte-identically, since Unix paths already contain no backslash, so it changes nothing for the recorded baseline and makes a Windows build reproduce it. That is small and I am inclined to do it, but it invalidates the fingerprint reasoning in the #[cfg(unix)] comment, so it gets its own commit and its own test rather than riding with the harness fixes.

The 276 residual misses you isolated — same file, neither separator form resolves, concentrated in 169 files — are a separate content divergence and I will not guess at them here; if the POSIX normalization lands and they persist, they become their own issue with the file list you already have.

Items 1–4 and the assertion are one train; I will note here when it lands.