Repository navigation
feat(ingest): --known-hashes -- re-emit only changed docs, so a re-ingest stops burning every embedding - #2
Merged
Conversation
…gest stops burning every embedding The emitted SQL opens with DELETE FROM documents WHERE repo = <tag>, and chunks cascade off documents. Chunks carry the embeddings, so editing one file and re-ingesting recomputes the entire corpus: on a 127-document set that is ~75 s of bge-m3 for a change worth two chunks. On anything larger it makes a routine refresh impractical, which pushes people towards not refreshing -- and a stale corpus fails silently, by answering from old text. documents.content_hash is already NOT NULL and written on every insert, but nothing reads it. This makes it load-bearing: pass --known-hashes with a `path<TAB>hash` TSV of what the DB holds and the ingester emits only new and changed documents, deletes changed and vanished ones by path, and leaves everything else -- chunks, embeddings, constraint links -- alone. Deliberately kept DB-free. The ingester still just writes a .sql file, because INSTALL.md's Kubernetes path depends on that (the store is not reachable from the laptop; the SQL is carried in). The diff input arrives as a file the operator produces however they can reach the store, so the incremental path works over kubectl exec and tunnels like everything else. Two interactions worth naming: - ci/freshness.py counts documents whose git_commit != HEAD as stale, and --mark writes status='stale'. Skipping unchanged rows would leave them at their old commit and report the whole corpus as stale after any commit. The incremental header refreshes git_commit for the repo first: the content IS current as of this commit, only the chunks were not rewritten. - Without --known-hashes the emitted SQL is byte-identical to before (verified by diffing the output of both versions over this repo), so existing callers are untouched. tests/test_incremental_ingest.py pins the contract from the outside, DB-free, and is wired into CI. It asserts the negatives that have no other symptom: no blanket DELETE, the unchanged document absent from the output, the changed and vanished ones inside the DELETE list, and the git_commit refresh present. A silent regression to full-replace would still produce a correct corpus -- it would only cost every embedding, which reads as "embedding is slow", so nothing else in CI would notice.
Recluse
added a commit
that referenced
this pull request
Sep 9, 2026
Skipping unchanged documents keeps their chunks, and chunks carry the embeddings, so a one-file edit no longer re-embeds the whole corpus. Verified locally before merge: the no-flag output is byte-identical over this repo, and the new contract test passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The emitted SQL opens with DELETE FROM documents WHERE repo = , and chunks cascade off documents. Chunks carry the embeddings, so editing one file and re-ingesting recomputes the entire corpus: on a 127-document set that is ~75 s of bge-m3 for a change worth two chunks. On anything larger it makes a routine refresh impractical, which pushes people towards not refreshing -- and a stale corpus fails silently, by answering from old text.
documents.content_hash is already NOT NULL and written on every insert, but nothing reads it. This makes it load-bearing: pass --known-hashes with a
path<TAB>hashTSV of what the DB holds and the ingester emits only new and changed documents, deletes changed and vanished ones by path, and leaves everything else -- chunks, embeddings, constraint links -- alone.Deliberately kept DB-free. The ingester still just writes a .sql file, because INSTALL.md's Kubernetes path depends on that (the store is not reachable from the laptop; the SQL is carried in). The diff input arrives as a file the operator produces however they can reach the store, so the incremental path works over kubectl exec and tunnels like everything else.
Two interactions worth naming:
tests/test_incremental_ingest.py pins the contract from the outside, DB-free, and is wired into CI. It asserts the negatives that have no other symptom: no blanket DELETE, the unchanged document absent from the output, the changed and vanished ones inside the DELETE list, and the git_commit refresh present. A silent regression to full-replace would still produce a correct corpus -- it would only cost every embedding, which reads as "embedding is slow", so nothing else in CI would notice.