[MXAntian]

Summary

Introduces a recall contract binding the memory pipeline to hard, inspectable limits. Recall was previously an unbounded FTS+vec+RRF fusion whose only output guarantees were "some rows, sorted by score" — no per-call ceiling on candidates or context tokens, no way to distinguish what the model saw from what the query asked for, and no defense against fabricated [id:N] citations.

Bridges 3 of the HMS/Shadow-Weave modules we agreed were worth porting (capacity hard-gate + full-pipeline trace + anti-hallucination allowlist) into mneme's existing hybrid retrieval, without hitting any of mneme's fallback contracts (no sqlite-vec / simple extension dependency introduced).

What this adds

Capacity budget (recall-contract.mjs)

• planRecallBudget(requested) — clamps requested → effective ≤ 20 and candidates ≤ 100. All recall paths route through it.
• enforceContextBudget(entries, {maxChars=12000}) — packs memories up to a hard character ceiling, records the stop reason (maxEntries / maxChars / natural).

Recall trace (recall-contract.mjs + migrations/010-add-recall-traces.sql + 2 MCP tools)

• Every recall creates a traceId and records: source, mode, query_hash (sha256, never raw), query_chars, requested/effective/candidate limits, kept_ids, per-step timing.
• recall_traces table stores hashes/counts/rowids only — query text and memory content deliberately not copied into a second persistence surface. Retention bounded by MNEME_RECALL_TRACE_RETENTION_DAYS (1-365) and MNEME_RECALL_TRACE_MAX_ROWS (100-100000).
• get_recall_trace(trace_id) MCP tool exposes the trace.

Anti-hallucination allowlist (recall-contract.mjs + MCP tool)

• stripHallucinatedMemoryIds(text, validIds) — regex-strips any [id:N] in generated text whose N is not in the trace's kept_ids.
• validate_memory_references(trace_id, text) MCP tool — a missing trace fails closed (strips all IDs), not open.

Query rewrite hook (query-rewrite.mjs + _noQueryExpansion internal flag)

• Ablation-controlled query expansion point wired into recallMemories.
• ⚠️ The seed rule (homegrown → garden/harvest/cooking/...) is proof-of-concept only. The intended shape is the hook + query.expand trace step, not the seed vocabulary. A follow-up PR should replace it with a generalized vocabulary. Landing the hook now unblocks the LongMemEval harness (test: LongMemEval retrieval-gate harness with stratified per-category sampling #8) that ablates against it.

FTS score normalization (recall-scoring.mjs)

• normalizedFtsScore(rank, maxMagnitude) — bounded [0,1] transform for bm25 rank, used inside the recall pipeline so FTS scores can be compared against vec similarity on the same scale.

Breaking change (v2.9)

recall_memory public limit is now hard-capped at 20 by the recall contract. Callers passing limit > 20 silently clamp — surfaced through the trace's effective_limit and (new) the CLI's capped field. package.json bumps 2.8.0 → 2.9.0 to signal this to caret consumers. Zod schema on the MCP tool declares .max(20) with an explanatory describe() so consumers see the ceiling in the schema itself.

Self-review round 1 (all fixes applied)

Ran a full fan-out self-check (Claude + Codex + Deepseek + hands-on repro) before requesting review. Findings and fixes in this commit:

P0

• Silent 20-cap was invisible: --recall --format json now returns requested_limit / effective_limit / candidate_limit / capped / trace_id alongside the hits. Zod schema on MCP tool tightened to .max(20).describe(...). package.json bumped 2.8.0 → 2.9.0. README (EN + ZH) documents the cap on the recall_memory row.
• stripHallucinatedMemoryIds one-space bypass: [ id:99] (single leading space after [) used to survive completely undetected — the regex required [ immediately followed by id. Regex now allows optional whitespace: \[\s*id\s*:\s*.../. Regression test added: leading-space variants like [ id:N] cannot bypass the allowlist.

P1

• access_count bumped for budget-dropped candidates: added _deferAccessBump opt to recallMemories / recallMemoriesHybrid; buildMemoryContext uses it and bumps only the rowids that survive the section-level context budget. Restores the "recall → access_count → ranked-higher" feedback-loop invariant this repo already carefully guards.
• Citation-contract header advertised budget-dropped IDs: renderContext now accepts advertisedIds — the budget-packing pass uses the upper bound (all candidate rowids) for honest char accounting; the final rendered header shows only the IDs actually injected. Trace chars field records the real post-rerender length.
• New stopwords blocked short conversational queries: used, recently, many, types, different were added to swallow one FTS false-negative pattern but also killed "what have I used recently" and similar. Reverted; comment in FTS_STOP_WORDS explains why. Regression test in query-rewrite.test.mjs proves "what have I used recently" still hits.
• New tests weren't in CI: .github/workflows/ci.yml now runs recall-contract.test.mjs and query-rewrite.test.mjs.
• Trace retention sweep hourly-only: added _writesSinceRecallTraceSweep counter — sweeps on either the hourly gate OR every 200 writes, so bursts cannot silently overshoot MNEME_RECALL_TRACE_MAX_ROWS intra-hour.

Migration 010 idempotency + old-DB upgrade were hands-on verified during self-check (built a DB on main, re-opened with this branch, confirmed IF NOT EXISTS upgrade + zero data loss). Trace persist failure path audited: no NaN, no content leak, fails open for reads.

Verification (per CONTRIBUTING — hands-on)

Post-fix suite:

suite	checks	result
node recall-contract.test.mjs	14 (was 13)	✅ all pass
node query-rewrite.test.mjs	6	✅ all pass
full upstream regression (meta-gate / anchor-pinned / hooks / memory-health / locations / recall-search-regressions / conversation-fallback-dedup / temporal-parser / temporal-recall)	155	✅ all pass

node --check on every touched file ✅.

Migration + compatibility

• migrations/010-add-recall-traces.sql — new table only, IF NOT EXISTS guarded, forward-only.
• recall_traces absence is not fatal — trace persist failure logs and continues.
• Backward-compatible otherwise: nothing in the public API rejects previously valid inputs. See "Breaking change (v2.9)" above for the limit cap.
• No sqlite-vec / simple extension dependency introduced.

Follow-ups (not in this PR)

• Replace query-rewrite.mjs seed rule with a proper generalized vocabulary (or jieba-driven expansion).
• LongMemEval retrieval-gate harness that exercises this contract → PR test: LongMemEval retrieval-gate harness with stratified per-category sampling #8.

Context

Comes out of 07-14 mneme audit vs 新智元 HMS/Shadow-Weave memory system, follows the merged #6 decay fix. Requesting @xiaomeng review since you own the audit.