Skip to content

feat(ingest): --known-hashes -- re-emit only changed docs, so a re-ingest stops burning every embedding - #2

Merged
Recluse merged 1 commit into
Recluse:mainfrom
boris-virto:feat/incremental-ingest
Sep 7, 2026
Merged

Recluse merged 1 commit into
Recluse:mainfrom
boris-virto:feat/incremental-ingest

Conversation

@boris-virto

Copy link
Copy Markdown
Contributor

The emitted SQL opens with DELETE FROM documents WHERE repo = , and chunks cascade off documents. Chunks carry the embeddings, so editing one file and re-ingesting recomputes the entire corpus: on a 127-document set that is ~75 s of bge-m3 for a change worth two chunks. On anything larger it makes a routine refresh impractical, which pushes people towards not refreshing -- and a stale corpus fails silently, by answering from old text.

documents.content_hash is already NOT NULL and written on every insert, but nothing reads it. This makes it load-bearing: pass --known-hashes with a path<TAB>hash TSV of what the DB holds and the ingester emits only new and changed documents, deletes changed and vanished ones by path, and leaves everything else -- chunks, embeddings, constraint links -- alone.

Deliberately kept DB-free. The ingester still just writes a .sql file, because INSTALL.md's Kubernetes path depends on that (the store is not reachable from the laptop; the SQL is carried in). The diff input arrives as a file the operator produces however they can reach the store, so the incremental path works over kubectl exec and tunnels like everything else.

Two interactions worth naming:

  • ci/freshness.py counts documents whose git_commit != HEAD as stale, and --mark writes status='stale'. Skipping unchanged rows would leave them at their old commit and report the whole corpus as stale after any commit. The incremental header refreshes git_commit for the repo first: the content IS current as of this commit, only the chunks were not rewritten.
  • Without --known-hashes the emitted SQL is byte-identical to before (verified by diffing the output of both versions over this repo), so existing callers are untouched.

tests/test_incremental_ingest.py pins the contract from the outside, DB-free, and is wired into CI. It asserts the negatives that have no other symptom: no blanket DELETE, the unchanged document absent from the output, the changed and vanished ones inside the DELETE list, and the git_commit refresh present. A silent regression to full-replace would still produce a correct corpus -- it would only cost every embedding, which reads as "embedding is slow", so nothing else in CI would notice.

…gest stops burning every embedding

The emitted SQL opens with DELETE FROM documents WHERE repo = <tag>, and chunks cascade off
documents. Chunks carry the embeddings, so editing one file and re-ingesting recomputes the
entire corpus: on a 127-document set that is ~75 s of bge-m3 for a change worth two chunks.
On anything larger it makes a routine refresh impractical, which pushes people towards not
refreshing -- and a stale corpus fails silently, by answering from old text.

documents.content_hash is already NOT NULL and written on every insert, but nothing reads it.
This makes it load-bearing: pass --known-hashes with a `path<TAB>hash` TSV of what the DB
holds and the ingester emits only new and changed documents, deletes changed and vanished
ones by path, and leaves everything else -- chunks, embeddings, constraint links -- alone.

Deliberately kept DB-free. The ingester still just writes a .sql file, because INSTALL.md's
Kubernetes path depends on that (the store is not reachable from the laptop; the SQL is
carried in). The diff input arrives as a file the operator produces however they can reach
the store, so the incremental path works over kubectl exec and tunnels like everything else.

Two interactions worth naming:

- ci/freshness.py counts documents whose git_commit != HEAD as stale, and --mark writes
  status='stale'. Skipping unchanged rows would leave them at their old commit and report the
  whole corpus as stale after any commit. The incremental header refreshes git_commit for the
  repo first: the content IS current as of this commit, only the chunks were not rewritten.
- Without --known-hashes the emitted SQL is byte-identical to before (verified by diffing the
  output of both versions over this repo), so existing callers are untouched.

tests/test_incremental_ingest.py pins the contract from the outside, DB-free, and is wired
into CI. It asserts the negatives that have no other symptom: no blanket DELETE, the
unchanged document absent from the output, the changed and vanished ones inside the DELETE
list, and the git_commit refresh present. A silent regression to full-replace would still
produce a correct corpus -- it would only cost every embedding, which reads as "embedding is
slow", so nothing else in CI would notice.
@Recluse
Recluse merged commit 305fca9 into Recluse:main Sep 7, 2026
Recluse added a commit that referenced this pull request Sep 9, 2026
Skipping unchanged documents keeps their chunks, and chunks carry the
embeddings, so a one-file edit no longer re-embeds the whole corpus.
Verified locally before merge: the no-flag output is byte-identical over
this repo, and the new contract test passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants