fix(brain): load vector index from disk on the search path - #546
Merged
Conversation
VECTOR_CACHE was only populated by embed_project (i.e. 'cora index'), so a fresh search process — 'cora brain' CLI or MCP brain_search — always saw an empty cache and the vector signal never fired; results silently degraded to FTS + graph only. Verified on develop and on the vecq PR branch with the same sandboxed e2e run. vector_search now lazy-loads the on-disk index once per process via ensure_vector_cache (double-checked read/write locking; a concurrent double load is idempotent). Load failures degrade gracefully with a warn log. Also guard dimension mismatch (embedding provider switched since the index was built) — previously a panic inside the backend. Note: the vecq backend still degrades to FTS-only in fresh search processes — that is the documented upstream limitation (vecq#32, no key serialization), tracked separately. Regression tests (tests/brain_search_process.rs): fresh-process brain search must emit the vector signal (default usearch backend), and brain.vector_store: vecq must produce a .vecq file — both run with isolated CODECORA_HOME sandboxes.
ajianaz
added a commit
that referenced
this pull request
Aug 31, 2026
…rain.vector_bits knob (#548) Upgrade the opt-in vecq store to vecq-core 0.3.0 and close #547. - Keyed persistence (upstream vecq#32 closed, format v1.3+): symbol-id keys survive reload. load_or_create_vecq now deserializes for real and keeps a healthy keyed index; legacy keyless files, corrupt files, and dim mismatches rebuild dirty so the next re-embeds. A dim guard is mandatory: add_keyed panics on wrong-dim vectors. - brain.vector_bits config (residual|4|5|6, lenient, default residual): residual is best recall@10 at 4-bit scan speed in a recall study on cora's own embeddings (ahead of plain 5-bit at 1k/5k/13k scales). A width change rebuilds the index once so the config takes effect; vecq_file_needs_rebuild is width-aware. - BrainConfig gets a manual Default (derived Default produced empty strings, inconsistent with the serde defaults). - Cross-project heal: the vector index is one shared file; a rebuild (width/dims/legacy/corrupt) now clears embed fingerprints for ALL projects, not just the active one, and the usearch dims-mismatch deletion path joins the same heal (pre-existing fingerprint leak). Tests: keyed roundtrip, legacy keyless rebuild, corrupt rebuild, dim-mismatch rebuild, width-switch rebuild, roundtrip across all four widths, lenient parsing, config merge. 947 pass incl. the fresh-process integration tests from #546. Closes #547 Co-authored-by: ajianaz <ajianaz@users.noreply.github.com>
This was referenced Aug 31, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The search path (
cora brainCLI and MCPbrain_search) never loaded the vector index from disk.VECTOR_CACHEwas only populated byembed_project(thecora indexpath), so every fresh search process saw an empty cache,vector_searchreturned nothing, and results silently degraded to FTS + graph signals only. Pre-existing ondevelop(default usearch backend) and on the vecq branch — verified with the same sandboxed e2e run (signals=ftsonly).Fix:
vector_searchnow lazy-loads the on-disk index once per process viaensure_vector_cache(double-checked read/write locking; concurrent double load is idempotent). Load failures degrade gracefully with a warn log and FTS fallback. Also guard dimension mismatch (embedding provider switched since the index was built) — previously a guaranteed panic inside the backend.Why
Without this, the vector signal — a third of the RRF fusion — was dead in every process that did not run the embed itself.
brain_searchquality silently regressed to FTS-only, and the vecq backend merged in #543 had no observable benefit on the search path.Testing
tests/brain_search_process.rs): fresh-processcora brainmust emit thevectorsignal with the default usearch backend;brain.vector_store: vecqmust produce a.vecqfile. Both use isolatedCODECORA_HOMEsandboxes. 2/2 pass.CODECORA_HOME):cora index→ fresh-processcora brain alphanow printssignals=fts,vector(and surfacesbetavia vector-only) where develop printedsignals=ftsalone.Note: the vecq backend still degrades to FTS-only in fresh search processes — upstream
vecq#32(no key serialization on reload) — tracked separately; the load path is backend-aware and will pick vecq up once upstream lands key serialization.Closes #545