Repository navigation
fix: delete files written by merge_insert analyze_plan - #9869
Merged
Merged
Conversation
analyze_plan executes the merge to collect metrics but never commits, so the data and deletion files it wrote were left unreferenced until orphan cleanup. Pull the uncommitted transaction off the executed write node and delete the new fragments' files, the patch files added to updated fragments, and any new deletion files. Closes #9862 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Esjxkkjia3FpyCY61Ao7AL
wjones127
marked this pull request as ready for review
October 10, 2026 05:13
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
With managed blobs enabled, Blob v2 payloads above the inline threshold are written as independent _blobs/<uuid>.blob objects that no fragment lists, so deleting the analyzed data files left them behind and repeated analyses kept accumulating payload bytes. BlobPreprocessor now records each managed payload path as it creates it into an optional BlobPayloadInventory, threaded through the crate-internal write chain. FullSchemaMergeInsertExec owns one, and analyze_plan deletes exactly what it recorded. Deleting by descriptor scan would be unsafe, since a reused Managed descriptor can point at a live payload. The in-place node refuses blob sources and never inserts, so it creates none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Esjxkkjia3FpyCY61Ao7AL
…-pk-flush # Conflicts: # rust/lance/src/dataset/blob/write.rs # rust/lance/src/dataset/versions/mod.rs # rust/lance/src/dataset/write.rs # rust/lance/src/dataset/write/insert.rs # rust/lance/src/dataset/write/merge_insert.rs # rust/lance/src/dataset/write/merge_insert/exec/write.rs
Xuanwo
approved these changes
Oct 28, 2026
Xuanwo
left a comment
Member
There was a problem hiding this comment.
Approved at f096c87 after merging the latest main and validating repeated analyze cleanup for data, deletion, and managed Blob payloads. Targeted Rust tests pass 5/5; fmt, clippy, and all applicable current-head CI pass; Gatekeeper approved this head.
This was referenced Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9862.
MergeInsertJob::analyze_plan/analyze_plan_batches/analyze_plan_providerrun the merge to collect metrics but never commit it. Until now, every file the run wrote stayed in storage with nothing referencing it until orphan cleanup removed it. That matters because LanceDB exposes analyze on remote tables, so repeated calls pile up orphaned files.Changes
analyze_plan_providerkeeps a handle to the executed plan. Afteranalyze_planreturns, it reads the uncommittedTransactionfrom the write node (FullSchemaMergeInsertExec,InPlaceMergeInsertExecorDeleteOnlyMergeInsertExec) and deletes what that transaction wrote. Cleanup runs whether or not the analysis succeeded.cleanup_data_fragments, so files routed to target bases and legacydata/<stem>/blob sidecars are covered too)RewriteColumnspatch file)_blobs/<uuid>.blobobjects that no fragment lists.BlobPreprocessornow records each one it creates, packed or dedicated, into an optionalBlobPayloadInventory. It records the path before opening the writer, so a partly uploaded object is covered too.write_fragments_internal_impl→versions::write_fragments→do_write_fragments_impl→WriterOptions). Existing callers passNone, and no public signatures change.FullSchemaMergeInsertExecowns an inventory, and analyze deletes exactly the recorded set, even if the run failed before building a transaction.RewriteColumnsrefuses any source that carries a blob column and never inserts rows, so it can't create payloads.analyze_planandexplain_planno longer say files are left behind.Testing
test_analyze_plan_removes_written_files(rstest, one case per write node: full-schema upsert, in-place update, delete-only) runs on a 2-fragment on-disk dataset. Each case checks that the set of files on disk is identical before and afteranalyze_plan_batches, and that the version and row count are unchanged.test_analyze_plan_removes_written_blob_payloads(packed and dedicated cases, adapted from the review reproducer) runs two analyses on a dataset with managed blobs. It checks that no_blobs/objects are left behind and that the committed blobs still read back. It also asserts that the fixture really stores payloads under_blobs/.cargo test -p lance --lib -- merge_insert dataset::blob dataset::write:: updater overlay: 937 passed.cargo fmt --allandcargo clippy -p lance --tests --benches -- -D warningsare clean.Not covered
If execution fails partway, the write node never stores a transaction, so the data files that run wrote can't be found this way. The write nodes already clean up after their own post-write failures, which covers most of that case. Managed blob payloads are deleted even in that case.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Esjxkkjia3FpyCY61Ao7AL