feat(search): MSCODEBASE_DENSE_OFF kill-switch + EXP-28 dense-off A/B

[ManSio]

What

Phase 3 of the migration, as a measurement — not a deletion. Adds MSCODEBASE_DENSE_OFF (default OFF) which skips the whole dense stage (query embedding + LanceDB vector search), plus the EXP-28 A/B that decides whether removal is ever justified, and EXP-29 which closes the one hypothesis EXP-28 had to leave open.

Verdict: dense removal NOT recommended. Quality is unchanged (net +1 of 33 = noise), speed is a stable ~7.5%, and — the real reason — LanceDB is not a vector index here but the chunk primary store.

EXP-28 — dense on vs dense off

Panel experiments/e28_denseoff/frozen/queries.jsonl, 33 cases, sha256 f31e594e…278cc8, union of three pre-existing frozen sets (4A f5, token_reduction, e2e_quality_search::CASES), deduped by gold file. 6 hypotheses frozen, 3 of them written to be expected to fail. 2 runs per arm, same session, same index; only env differs.

arm	hit@1	hit@5	errors	mean_ms (run1/run2)
A dense ON	11/33 (33.3%)	15/33 (45.5%)	0	3359 / 3534
B dense OFF	12/33 (36.4%)	16/33 (48.5%)	0	3163 / 2492

Per-case: 2 better / 1 worse / 30 same. H1/H2/H6 refuted (as intended), H4 confirmed, H3 PARTIAL, H5 untested — the panel had no symbol/NL labelling, and labelling cannot be added after the run (§17).

Harness proven, not assumed: a _vector_search_async call counter goes 1 → 0 between arms with the embedder alive. Without that check the A/B would have been empty.

EXP-29 — the symbol/NL split that closes H5

Labelling rule frozen first (e29_label.py: symbolic := snake_case | CamelCase | pathy(:: / slash / .method()) | single token), written into the frozen panel (queries_labeled.jsonl, sha256 7639ba1b…, symbolic 15 / NL 18), then both arms twice.

arm	NL hit@1 (18)	symbolic hit@1 (15)	errors	mean_ms
A dense ON	3 (16.7%)	8 (53.3%)	0	2559 / 2555
B dense OFF	5 (27.8%)	7 (46.7%)	0	2375 / 2354

NL: 2 better / 0 worse. Symbolic: 0 better / 1 worse (TR-P2). H1 CONFIRMED (symbolic loses exactly 1 of 15 — inside the declared tolerance), H2/H3 refuted as expected, H4 CONFIRMED, H5 CONFIRMED against expectation.

Mechanism: both improvements are NL queries where dense pulled top-1 into a document (docs/adr/0003-verify-on-read.md, src/core/multi_project_searcher.py). The single regression is lexical (docs/zh/SEARCH_PIPELINE.md instead of engine.py). Subset sums reproduce EXP-28 exactly (A 8+3=11, B 7+5=12) — same panel split, not an independent experiment.

CORRECTED (no silent edit)

EXP-28 published an 18.0% speedup. EXP-29's two runs give 7.2% and 7.9% — the old figure was inflated by one noisy run (21% spread vs 0.2% here). EXP-28's H3=PARTIAL verdict stands, but the 18% number is retired; the live figure is 7.5%. Logged as a new entry with a supersede reference, per §11/§19.11.

The finding that matters more than the numbers

Of 52 table-API call sites in 13 src/ files, only 5 are vector search. The rest is chunk storage: bm25.py:93 builds the BM25 index from table.to_pandas(), db_writer/indexer/indexer_table write and delete chunks, file_move_manager + index_guard do version()/where() scans for meta-patching and repair. Deleting LanceDB would be a store migration, not a cleanup.

Harness incident (caught, fixed)

The first EXP-29 run from the worktree printed hit1=0/33 with rc=0 in 167 ms — a silent zero on an empty index, because the index is per-project-path and the worktree has none. Guard added: all-empty input exits 2 with an explicit message, and the project root is an explicit manifest parameter. That run is not counted. Separately, test_no_personal_paths caught a machine-specific absolute path leaking into EXPERIMENTS_LOG.md — replaced with a placeholder before commit.

Safety

• Default OFF: the prod path is byte-identical without the flag (one added condition at engine.py:936).
• Tests: 3 flag tests + adjacent suites 29/29, ruff clean.
• KNOWN_ISSUES.md deliberately untouched (another session has uncommitted edits there) — proposals go to experiments/e28_denseoff/PENDING_LEDGER.md.

Verification

Clean state at 3813561: Passed: 2052 / Failed: 0 — CLEAN STATE VERIFICATION: PASSED.

Notes

• Stacked on feat(search): on-demand cosine rerank path behind flag (Phase 2) #74 (Phase 2), which is stacked on feat(index): E26 file-level chunk augmentation behind flag (default OFF) #73 (Phase 1). Merge order: fix(search): graceful fast-mode fallback when embedder is down #72 → feat(index): E26 file-level chunk augmentation behind flag (default OFF) #73 → feat(search): on-demand cosine rerank path behind flag (Phase 2) #74 → this.
• Stacked PRs do not trigger the CI workflow (it listens to pull_request → main only); the clean-state run above is the equivalent evidence.
• No reindex required.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: bc5f8060-fe25-48e8-9c5e-98ff23ac5261

---

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}