aft-search bench: the harness indexes its own answer key; normalize_result_path breaks on Windows verbatim paths

[Qiiks]

Harness integrity: the fusion-quality bench indexes its own answer key, and normalize_result_path cannot handle Windows verbatim paths

Two defects in benchmarks/aft-search/, found while running the rerank probe in #322. Filing separately as you asked. Neither is a request for a fix to the engine — both are bench-side.

1. The harness indexes its own answer key

run-fusion-quality walks --project-root and AFT indexes whatever it finds. The benchmark's own fixture and result files contain every fixture's query and its expected_top_files:

• benchmarks/aft-search/fixtures.json — tracked
• benchmarks/aft-search/identifier-fusion-fixtures.json — tracked
• benchmarks/aft-search/baseline.json — tracked
• benchmarks/aft-search/results/search-fusion-quality.json — tracked, and the harness's own default --out

The harness excludes these from its scratch exact-match oracle, and the comment says exactly why:

# Exclude benchmark/report artifacts from the bench-only exact-match oracle so
# the query strings in fixture/result/report files do not become self-fulfilling
# exact hits.

Nothing applies that exclusion to AFT's index. There is no .aftignore at the benchmark root, so the measured system ingests the answer key and returns it. Direct AFT call, no harness in the path (probe-answerkey.py):

=== 'subagent_type' -> 10 results
  r1 [exact]   .alfonso/reports/search-fusion-quality.md
  r2 [exact]   benchmarks/aft-search/baseline.json                   <== ANSWER KEY
  r3 [exact]   benchmarks/aft-search/fixtures.json                   <== ANSWER KEY
  r4 [exact]   benchmarks/aft-search/identifier-fusion-fixtures.json <== ANSWER KEY
  r5 [lexical] assets/aft.schema.json

Identical shape for aft_safety_history and LEXICAL_ONLY_SCORE_CEILING. 33 of 41 fixtures had at least one answer-key file in their top-10, all from the exact lane.

Why it matters: it deflates the baseline

Answer-key files crowd the top of the list and push the correct file down, so current measures worse than it is. Re-ranking the recorded pools with the key files removed:

suite	current with key files	current without
existing	R@1 0.286 / MRR 0.343	R@1 0.393 / MRR 0.423

The direction is the dangerous one: a suppressed baseline makes every rerank candidate look better than it is. My own probe gain shrank from ~+0.25 MRR to ~+0.15 MRR once the baseline was measured correctly.

Fix

Add .aftignore at the benchmark root (AFT honors it — crates/aft/src/callgraph.rs:3502, context.rs:3353) and rebuild:

benchmarks/aft-search/fixtures.json
benchmarks/aft-search/identifier-fusion-fixtures.json
benchmarks/aft-search/baseline.json
benchmarks/aft-search/external-fixtures.json
benchmarks/aft-search/real-query-baseline.json
benchmarks/aft-search/results/
.alfonso/

Verified effect: search index 3,526 → 3,421 files; answer-key rows in top-10s 33 → 0.

Checked since, and the real-query gate is NOT affected. I flagged this as likely and it was wrong, so correcting it here rather than leaving a scare standing. Two independent reasons, both verified:

1. The manifest is not in the indexed tree. run_real_query.py:549 indexes a runtime_evidence_tree(provisioned_tree) — benchmarks/aft-search/.bench/repos/aft-evidence-<sha>/. That tree contains fixtures.json, identifier-fusion-fixtures.json and baseline.json, but not real-query-manifest.json and not real-query-baseline.json (absent on disk). The manifest holds the real-query query/opened_file pairs, so the gate's own answer key is outside the corpus it measures.
2. The query sets are disjoint anyway:

fusion fixture queries : 39
real-query included rows: 43 (41 distinct queries)
overlap with the indexed fusion answer keys: 0

opened_file values are ordinary source paths (Cargo.lock, etc.) that belong in the corpus legitimately — that is the retrieval target, not a leak.

So defect 1 is scoped to run-fusion-quality; the per-mechanism baseline that gate produces is not suppressed this way.

One narrow footnote that does carry over. The evidence tree does index benchmarks/aft-search/fixtures.json and identifier-fusion-fixtures.json, so if a future real-query row ever reused a fusion-fixture query verbatim, that row would inherit the same self-fulfilling exact hit. It is zero today, and cheap to keep at zero by adding the same .aftignore entries to the evidence tree — worth doing deliberately rather than relying on the two sets happening not to intersect.

2. normalize_result_path cannot handle Windows verbatim paths

run.py:

def normalize_result_path(raw_path: str, project_root: Path) -> str:
    path = Path(raw_path)
    if path.is_absolute():
        try:
            return path.resolve().relative_to(project_root).as_posix()
        except ValueError:
            return path.as_posix()
    return path.as_posix()

AFT returns files under the \\?\C:\... verbatim form on Windows. Path.resolve() preserves that prefix, so relative_to() raises ValueError and the function returns the raw verbatim string. Every fixture then compares a relative expected path against an absolute verbatim one, never matches, and current scores a clean 0.000 across all 41 fixtures on a perfectly healthy index (33,747 entries) — with no error, no warning, and a well-formed result file.

That is the worst failure shape: a plausible-looking zero. It cost me a full run before I caught it, and it nearly went to the maintainer as a result.

for prefix in ("\\\\?\\", "//?/"):
    if raw.startswith(prefix):
        raw = raw[len(prefix):]
        break

Offline-fusion fixtures are what Windows contributors exercise, and this makes them unusable without an out-of-tree shim.

Method note (may save you an hour)

Driving run-fusion-quality from an external script, patches must be installed before importing the harness: it does from run import normalize_result_path, which binds the name at import time. Patching run.normalize_result_path afterwards leaves the harness holding the original — same silent 0.000.

[aft-alfonso]

Both confirmed from the harness source, and the first one is the more important finding of the pair because it moved a number that had already been cited in a design decision.

On the answer key: the exclusion at line 41 was written for the scratch oracle and never applied to the corpus the measured system indexes, exactly as you say. The fix is the .aftignore you list at the benchmark root, plus the same entries in the evidence tree that run_real_query.py provisions — you found that one zero today, and I would rather it be zero by construction than by the two query sets happening not to intersect. The corrected fusion baseline (R@1 0.286 → 0.393 without the key files) supersedes the one recorded on this thread's parent; I will re-record it when the ignore lands so the committed baseline.json stops carrying the deflated number.

On normalize_result_path: the prefix strip is right, and I am putting it at the one place the verbatim form enters the harness rather than at each caller, for the import-binding reason you documented. I am also making a zero-match result loud — a well-formed report with every fixture at None on a non-empty index is the failure shape you described, and the harness should refuse to write it rather than let it look like a regression. That covers the same defect as #326 item 2, so the two land together.

Your method note about import-time binding goes into the harness README; it is the kind of thing that costs every new contributor an hour.