Skip to content

fix: delete files written by merge_insert analyze_plan - #9869

Merged
Xuanwo merged 4 commits into
mainfrom
claude/optimistic-cannon-efpzw4
Oct 28, 2026
Merged

Xuanwo merged 4 commits into
mainfrom
claude/optimistic-cannon-efpzw4

Conversation

@wjones127

@wjones127 wjones127 commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Closes #9862.

MergeInsertJob::analyze_plan / analyze_plan_batches / analyze_plan_provider run the merge to collect metrics but never commit it. Until now, every file the run wrote stayed in storage with nothing referencing it until orphan cleanup removed it. That matters because LanceDB exposes analyze on remote tables, so repeated calls pile up orphaned files.

Changes

  • analyze_plan_provider keeps a handle to the executed plan. After analyze_plan returns, it reads the uncommitted Transaction from the write node (FullSchemaMergeInsertExec, InPlaceMergeInsertExec or DeleteOnlyMergeInsertExec) and deletes what that transaction wrote. Cleanup runs whether or not the analysis succeeded.
  • What gets deleted, by comparing the transaction against the read version's manifest:
    • every file of a new fragment (reuses cleanup_data_fragments, so files routed to target bases and legacy data/<stem>/ blob sidecars are covered too)
    • data files an updated fragment has that the original didn't (the in-place RewriteColumns patch file)
    • deletion files that differ from the original fragment's
  • Managed Blob v2 payloads. With managed blobs enabled, payloads above the inline threshold are written as independent _blobs/<uuid>.blob objects that no fragment lists.
    • BlobPreprocessor now records each one it creates, packed or dedicated, into an optional BlobPayloadInventory. It records the path before opening the writer, so a partly uploaded object is covered too.
    • The inventory is threaded through the crate-internal write chain only (write_fragments_internal_impl → versions::write_fragments → do_write_fragments_impl → WriterOptions). Existing callers pass None, and no public signatures change.
    • FullSchemaMergeInsertExec owns an inventory, and analyze deletes exactly the recorded set, even if the run failed before building a transaction.
    • Nothing is deleted by scanning descriptors, because a reused Managed descriptor can point at a live payload.
    • The in-place node gets no inventory: RewriteColumns refuses any source that carries a blob column and never inserts rows, so it can't create payloads.
  • A failed delete is logged, not returned. The files are unreferenced either way, and orphan cleanup still catches anything left behind.
  • Doc comments on analyze_plan and explain_plan no longer say files are left behind.

Testing

  • test_analyze_plan_removes_written_files (rstest, one case per write node: full-schema upsert, in-place update, delete-only) runs on a 2-fragment on-disk dataset. Each case checks that the set of files on disk is identical before and after analyze_plan_batches, and that the version and row count are unchanged.
  • test_analyze_plan_removes_written_blob_payloads (packed and dedicated cases, adapted from the review reproducer) runs two analyses on a dataset with managed blobs. It checks that no _blobs/ objects are left behind and that the committed blobs still read back. It also asserts that the fixture really stores payloads under _blobs/.
  • With the respective cleanup disabled, every case of both tests fails.
  • cargo test -p lance --lib -- merge_insert dataset::blob dataset::write:: updater overlay: 937 passed.
  • cargo fmt --all and cargo clippy -p lance --tests --benches -- -D warnings are clean.

Not covered

If execution fails partway, the write node never stores a transaction, so the data files that run wrote can't be found this way. The write nodes already clean up after their own post-write failures, which covers most of that case. Managed blob payloads are deleted even in that case.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Esjxkkjia3FpyCY61Ao7AL

analyze_plan executes the merge to collect metrics but never commits, so
the data and deletion files it wrote were left unreferenced until orphan
cleanup. Pull the uncommitted transaction off the executed write node and
delete the new fragments' files, the patch files added to updated
fragments, and any new deletion files.

Closes #9862

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Esjxkkjia3FpyCY61Ao7AL
@github-actions github-actions Bot added the bug Something isn't working label Oct 10, 2026
@wjones127
wjones127 marked this pull request as ready for review October 10, 2026 05:13

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Oct 10, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Oct 19, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Oct 19, 2026
With managed blobs enabled, Blob v2 payloads above the inline threshold
are written as independent _blobs/<uuid>.blob objects that no fragment
lists, so deleting the analyzed data files left them behind and repeated
analyses kept accumulating payload bytes.

BlobPreprocessor now records each managed payload path as it creates it
into an optional BlobPayloadInventory, threaded through the crate-internal
write chain. FullSchemaMergeInsertExec owns one, and analyze_plan deletes
exactly what it recorded. Deleting by descriptor scan would be unsafe,
since a reused Managed descriptor can point at a live payload. The
in-place node refuses blob sources and never inserts, so it creates none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Esjxkkjia3FpyCY61Ao7AL
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 19, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The Managed Blob retention issue is fixed: analysis now removes newly created payloads while preserving reused references.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 19, 2026
…-pk-flush

# Conflicts:
#	rust/lance/src/dataset/blob/write.rs
#	rust/lance/src/dataset/versions/mod.rs
#	rust/lance/src/dataset/write.rs
#	rust/lance/src/dataset/write/insert.rs
#	rust/lance/src/dataset/write/merge_insert.rs
#	rust/lance/src/dataset/write/merge_insert/exec/write.rs
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 28, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The merge preserves cleanup ownership: analysis removes newly written files and payloads while keeping existing and explicitly shared Managed Blob references readable.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 28, 2026

@Xuanwo Xuanwo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at f096c87 after merging the latest main and validating repeated analyze cleanup for data, deletion, and managed Blob payloads. Targeted Rust tests pass 5/5; fmt, clippy, and all applicable current-head CI pass; Gatekeeper approved this head.

@Xuanwo
Xuanwo merged commit 35a81a0 into main Oct 28, 2026
43 of 44 checks passed
@Xuanwo
Xuanwo deleted the claude/optimistic-cannon-efpzw4 branch October 28, 2026 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

merge_insert analyze_plan leaves orphaned data files

3 participants