Discussion: where would a post-fusion rerank hook for semantic-gap queries be welcome?

[Qiiks]

Summary

A question before any code: the multi-lane search (exact/lexical/semantic/anchored + evidence tiers + kind priors) is already well-engineered, and I'm not proposing to replace any of it. Is there interest in an optional, calibrated rerank stage above the fusion, aimed specifically at semantic-gap queries — and if so, where would it belong architecturally?

The gap I measured

AFT's kind priors and RRF fusion are uniform across definition kinds (semantic_kind_multiplier gives every Function/Class/Struct 1.08 under NaturalLanguage strength, mod.rs:4423), so when a query's wording shares no tokens with the right answer's name, the fused order falls back to raw cosine order. Two failure surfaces I hit in practice:

• Intent-vs-mention traps: processRefund (returns money) vs cancelPurchase (merely mentions refund in its docstring) — cosine ranks them by embedding similarity, not by which one does the thing the query asks.
• Paraphrased intent: "where is money returned when a purchase is called off?" shares zero tokens with processRefund.

A small live probe (7-candidate shortlists, hosted decision model scoring each candidate's relevance with typed probability judgments): 4/4 top-1 vs raw cosine 1/4 on paraphrase queries, and clean intent separation (0.930 vs 0.050 for "actually returns money"). Caveat, stated plainly: this was measured against a raw-cosine shortlist, not against AFT's real fused output — the local semantic engine was unavailable during testing. I'm presenting the mechanism, not claiming AFT's fusion underperforms.

What the hook would look like

• Placement: after freeze_non_exact_scores produces the fused list — a post-fusion stage that may reorder (or annotate) the top-N, never add/remove candidates. Exact-tier results stay score-free per R16 and are never touched.
• Opt-in + fail-open: off by default; if the external judge errors or times out (~1s budget), the fused order passes through untouched. Zero added latency when disabled.
• No new failure mode on the hot path: the judge call would be async/bounded, and results carry an annotation when reranked so provenance stays honest (source/rank_score fields already exist for exactly this kind of metadata).
• Config: something like search.rerank: "off" (default) with the judge provider configurable.

Honest counter-case (why this might be a "no")

• The fused system already handles the common cases via exact/lexical lanes; semantic-gap queries may be rare enough not to justify the dependency.
• An external scoring endpoint is a new vendor dependency AFT (local-first, fast, deterministic) may not want at all.
• A local cross-encoder could do this dependency-free, at the cost of shipping/benchmarking a model.

Why I'm asking rather than PRing

I've had good experiences contributing here (#267, #300) — both landed as discussions first. This one touches the search scoring path, which is clearly carefully tuned (pinned plan table, R16 score-free exact tier, cap protection), so I'd rather hear where (or whether) the maintainers think a rerank hook fits before writing any of it.

Happy to share the full probe methodology and numbers, or to prototype behind the default-off flag if there's interest.

[aft-alfonso]

Thanks for asking first; this is the right shape for a change on the scoring path. Short answer: yes, there is a place for it, and there is a bar it has to clear before it can be on by default or even land behind a flag. Both below.

Where it belongs

Not a new hook. The engine already has an extension seam that later ranking work is meant to override one method at a time: SearchExtensions in crates/aft/src/commands/semantic_search/extensions.rs (classify / variants / sample_readiness / plan / execute_lane, with DefaultSearchExtensions and the installed B2SearchExtensions in crates/aft/src/search_b2/mod.rs). A rerank stage would be one more method on that trait with a pass-through default, so the base engine stays buildable and testable without it.

The placement you describe, after freeze_non_exact_scores in scoring.rs, is right, with one constraint that follows from the paging contract: pages are block-frozen (blocks.rs), so offset pages are provably equal to a single deeper retrieval and the ranked tuple is byte-identical at every depth. A reranker that may reorder differently on page 1 than it did on page 0 (a judge that timed out on one call and answered on the next) breaks that. So the stage has to be deterministic per (query, snapshot generation, admitted block): either its verdicts are cached under that key for the life of the query, or it is applied only to the first block and annotated as such. Exact-tier rows stay untouched and score-free, as you already have.

Provenance goes where you expect: a reranked row carries a source/rank_score annotation, and lane_positions keeps the pre-rerank position so the effect is auditable.

The bar

Ranking changes here are gated by a benchmark, not by review. The families live under benchmarks/aft-search/: concept, exact-recall, and the real-query manifest (rows minted from production telemetry, scored by the rank of the file the agent actually opened, run in CI as search-quality-gate). A slice that moves ranked rows lands only with a descriptor naming which measured failure mechanism it improves, and MRR@10 on that mechanism has to strictly improve with nothing else regressing.

Two data points from those families that bear on your hypothesis:

• The mechanism census behind the current engine did not surface "semantic gap" as a distinct failure class. Ranked real mechanisms were wrong lane on natural-language queries, index still building, top-K cut, phrase present but not surfaced, token variants, scope mismatch. The median query is two words; 31% are a single identifier, 27% code literals, 28% natural language.
• The dense lane does admit off-topic rows: after the model-card query instruction landed, the share of dense top-10 rows sharing zero content tokens with the query fell from 59% to 49%, with real-query MRR moving 0.182 → 0.197. That band is exactly where a reranker would earn its keep, and it is measurable today.

So the probe to run is your mechanism against the real fused output, on the real-query rows, not against a raw-cosine shortlist. If you want to prototype behind search.rerank: "off", the harness will score it: benchmarks/aft-search/README.md (the "Real-query quality gate" section; scripts/telemetry/cost-gate.sh --search-quality --mode evaluate is the entry point). Paraphrase rows that share no tokens with the target are not in the manifest today because the census did not produce them; if your probe queries are real ones you have hit, they can be labelled and added as a fixture family, which is how the last three mechanisms were admitted.

On the dependency

A hosted judge on the query path is not something AFT will turn on by default: local-first, deterministic, no vendor in the hot path. As an opt-in provider it is acceptable if it reuses the existing backend shape (semantic.backend already carries openai_compatible / ollama endpoints, timeouts and budgets) rather than adding a second credential surface. A local cross-encoder is the more likely thing to become default if the numbers hold, and the same harness would decide that.

Happy to look at the probe methodology and numbers here before any code.

[Qiiks]

Thank you for the thorough answer — this is exactly the framing I needed, and asking first was clearly right.

To confirm I've understood the constraints correctly before running anything:

1. Seam: rerank = one more SearchExtensions method with a pass-through default, not a new hook. That keeps the base engine buildable/testable standalone — agreed, that's the right shape.
2. Paging determinism: since pages are block-frozen, the stage must be deterministic per (query, snapshot generation, admitted block) — verdicts cached for the query's life, or first-block-only with annotation. Understood; a judge that times out on page 0 and answers on page 1 would break the offset-equals-deeper-retrieval contract, so this has to be designed in from the start, not patched.
3. Census correction accepted: "semantic gap" is not a surfaced failure class, and my probe was against a raw-cosine shortlist, not the real fused output — so it's hypothesis-generating, not evidence. The band you name (dense-lane zero-content-token rows: 59%→49%, MRR 0.182→0.197) is the measurable target, and the bench families already have the machinery to test it.
4. Bar: slice descriptor naming the measured mechanism, MRR@10 strictly improving on that mechanism with nothing else regressing, run through the search-quality-gate.
5. Dependency: hosted judge never default-on, and any opt-in must reuse the semantic.backend shape (endpoint/timeout/budget via the existing credential surface) rather than adding a second one. A local cross-encoder staying the more likely default candidate if numbers hold is a constraint I'll design around.

Next step: I'll run the probe against the real fused output — offline reranker shapes first via run-fusion-quality (on fixtures.json / identifier-fusion-fixtures.json), then the real-query gate (cost-gate.sh --search-quality over real-query-manifest.json rows) — and report methodology + numbers here before writing any code. If the probe holds up, I'd label the paraphrase-style queries I've actually hit as a fixture family proposal, justified by the mechanism (zero-content-token dense rows in the real-query manifest), not by anecdotes — my personal queries are an n-of-one sample.

---

Full map of where else the same decision model was surveyed in this codebase — since the rerank question came out of a broader audit, listing the other points here so you can steer which (if any) deserve their own issue rather than me filing them cold:

• Tier-6 compressor fallback — the 5-tier compress pipeline runs synchronously on the BgTaskRegistry watchdog thread (context.rs:2345-2348), so an inline network call there is off the table. Workable shapes: a tier-6 judge invoked only when tiers 1–5 leave output over the final cap, or async post-hoc refinement at the next idle moment. Measured judgment quality on a realistic failing-test log: picked the root-cause error line over 5 distractors (conf 0.98), keep-error-location 0.750, drop-compile-spam 0.630; 1.04s, 1,318 input tokens. Chunking mandatory regardless (32k-token state ceiling on the judge).
• Confidence-floor recalibration — HIGH_CONFIDENCE_COSINE_FLOOR 0.60 is documented as deliberately high because "MiniLM scores are uncalibrated, under-trigger" (mod.rs:4559). A calibrated judge could replace the floor; same seam as the rerank hook, and it would need fused-output evidence first. Enumerated, not measured.
• results_are_low_confidence gate (mod.rs:4565) — same seam. Enumerated, not measured.
• rank-0 snippet expansion (should_expand_rank0_snippet, mod.rs:5019) — judge "is rank-0 actually the answer". Enumerated, not measured.
• Callgraph hub-summary selection (HUB_SUMMARY_THRESHOLD/LIMIT) — surveyed and rejected: structural selection, not semantic judgment.

My read: the rerank hook is the only one with a measurable failure band already instrumented in your bench families, so it's the right first target — the tier-6 point is the one I'd most expect to be useful (it's the only one where the judge does something the existing tiers structurally can't: semantic root-cause selection in arbitrary tool output), but it has no benchmark family today. Would you want a separate issue for tier-6 with a proposed descriptor, or is it noise until the rerank question resolves?

For reference, the decision-model projects this came out of: fast-jev-compaction (transcript mutation at Claude Code's compaction event — fits their contract, not this engine's) and jev-sift (batched relevance screening — the same contract as the rerank hook).

[aft-alfonso]

All five constraints read back correctly, and the probe order you propose is the right one: offline fusion fixtures first, real-query gate second, fixture-family proposal only if the mechanism holds. One correction and one addition before you spend the time.

The correction, on the band. 49% of dense top-10 rows sharing zero content tokens with the query is not 49% wrong. It is the dense lane doing its job in a fused pipeline: a row that shares no surface token can still be the answer (that is why the lane exists), and the fusion demotes the ones that are not. So the target is not "shrink the band" — it is: among rows in that band, does your stage move the file the agent actually opened up, without moving any other row's file down. That distinction decides what your probe measures. The real-query rows carry the opened file, so it is measurable; a band-shrinkage number is not evidence.

The addition, on determinism. Caching verdicts per (query, snapshot generation, admitted block) is necessary but not sufficient: the cache has to be keyed on the admitted set, not on the query, because two pages of the same query admit different blocks and a verdict computed over block 0 is not defined for block 1. Either the stage runs once over the admitted set at retrieval depth and its output is frozen for every page of that snapshot, or it is first-block-only and annotated. The second is cheaper and I would start there. Also note that a timeout is not a neutral failure: a judge that answers on one page and times out on the next produces two different orders under one snapshot, so on timeout the stage must fall back for the whole query, not for the page — a decision you record once and reuse from the cache.

On tier-6. File it, and file it separately — it is a genuinely different claim, and it does not have to wait for the rerank question. But the framing you gave is the thing to fix before it can be judged: the compressor's job is not "pick the root-cause line", it is "produce output the next agent turn can act on", and no benchmark family exists because nobody has defined what a correct compression is. That is the work the issue should propose — a family (input transcript, cap, and a check that the kept lines contain the ones a later turn actually referenced), not a judge. With that family in place, a tier-6 judge is measurable and so is the far cheaper question of whether tiers 1-5 are leaving anything important out at all. Without it, a judge on the watchdog thread is a network call in the compression path with no way to show it helped.

Two of your enumerated points I would set aside rather than file: HIGH_CONFIDENCE_COSINE_FLOOR and results_are_low_confidence are not tuning knobs waiting for a better estimator — confidence is now a rule with a calibrated margin and a parity split behind it, and replacing it with a judge's probability means replacing a thing with a measured precision gap by a thing without one. Rank-0 snippet expansion is a context-cost tradeoff, not a relevance judgment; the cheaper experiment there is to measure how often the expanded snippet is followed by a read of the same file, which needs no judge at all.

Finally, a practical note on running the gate: the real-query corpus provisions from local clones pinned by sha (.bench/repos), so a first run does the checkout before it measures anything. And when you report numbers, report them as the gate does — per mechanism with the descriptor class, not as an aggregate; an aggregate that improves while one mechanism regresses is exactly what the per-mechanism rule exists to catch.

[Qiiks]

Consolidated result. This replaces eight earlier comments of mine on this thread (the probe run, its successive corrections, and the harness-defect reports) so the finding reads once instead of accumulating. The maintainer's two comments above are untouched, and my constraints reply above them still stands as the pre-probe agreement.

Where it landed

suite	candidate	R@1	MRR
existing (natural-language)	current	0.393	0.482
existing	jev_judge	0.536	0.628–0.629
identifier	current	0.385	0.404
identifier	jev_judge	0.231–0.308	0.312–0.367
all	current	0.390	0.457
all	jev_judge	0.439–0.463	0.528–0.546

Two independent runs against a clean index, baseline bit-identical both times. Judge verdicts: 9 improved / 4–5 regressed / 27–28 unchanged, every "improved" genuinely inside the top-10.

Verdict: the stage as designed fails your bar. It gains +0.147 MRR on the natural-language mechanism and regresses on identifier (R@1 0.385 → 0.231–0.308). Per the per-mechanism rule, a stage that improves one mechanism while regressing another cannot land as a drop-in — it needs a shape gate, not a bare stage.

Your constraints, measured

• Determinism. The judge is stochastic: pool identical 41/41 and baseline identical, yet 6/41 fixtures change rank between runs. So the verdict cache is not an optimization, it is the only thing making the stage admissible — and the identifier regression survives every run, which is the strongest form that negative result could take.
• Opened-file movement, not band shrinkage. That is what the table reports: rank of the file the agent actually opened. I did not measure band width.
• Cost. rerank p95 5,199–7,272 ms, p50 ~5 s. That is ~5x the ~1s the proposal itself named, and it decides against the interactive case — this is batch territory.
• Cache key on the admitted set, not the query. Understood and not contested; the probe measured the mechanism, not a shipping configuration.

Two defects in the harness, found while getting these numbers

Filed separately as #325, both affecting measurement rather than ranking:

1. The bench indexes its own answer key. No .aftignore at benchmarks/aft-search, and fixtures.json / identifier-fusion-fixtures.json / baseline.json / results/ (all tracked) contain every fixture's query and expected_top_files. AFT returns them as exact hits at rank 2–4; 33/41 fixtures had one in top-10. The harness's own comment at line 41 says these are excluded from its scratch oracle "so the query strings … do not become self-fulfilling exact hits" — that exclusion was never applied to AFT's index. Effect: it deflates the baseline, which inflates every candidate's apparent gain. I first reported this probe as ~+0.25 MRR; on a clean index it is ~+0.15.
2. normalize_result_path cannot handle Windows verbatim paths. AFT returns \\?\C:\…; resolve().relative_to() raises and the fallback returns the absolute verbatim string against a relative expectation, so every fixture misses and current scores a silent 0.000 on a healthy 33,747-entry index. This one nearly reached you as a result.

I also flagged the real-query gate as likely affected by (1). It is not — corrected on #325: its indexed tree is the provisioned evidence tree, which does not contain the manifest, and the fusion and real-query query sets have 0 overlap. The per-mechanism baseline that gate produces is not suppressed this way.

What I am not claiming

• No default-on path. Vendor-in-the-hot-path stands, and a local deterministic cross-encoder remains the shape that could actually be default.
• No real-query number. That gate is Unix-only in practice: chunk text bakes the OS-native relative path into the header, the pack and baseline are Unix-captured, so on Windows the index build cannot complete. Upstream documents this deliberately (semantic_index.rs:8143); the gate surfaces it as an opaque 422 vector_missing:<digest> rather than saying so, which is filed as search-quality gate is unrunnable on Windows: select() on a pipe, verbatim paths (silent 0.000), autocrlf hash pin, and ORT dll auto-detect #326 along with four genuine Windows harness defects.
• No fix proposed for the separator behaviour — it is documented design with a stated rationale, and re-deciding it is not mine.

Artifacts: fq-clean.json / fq-clean2.json (+ .tsv), polluted runs retained for the delta comparison, probe-answerkey.py (reproduces the leak directly against AFT).