Repository navigation
Conversation
merge_existing_index_segments keeps a segment's coverage of a fragment that compaction produced after the segment was built only if it can trace the fragment back to the segment's build snapshot through the fragment reuse index. It looked back one group, so a fragment made by compacting an earlier compaction's output had a source the snapshot never held: the fragment was taken as changed and left out of the merged coverage, and committing the merge over the segment it replaces was then refused for orphaning it. Each source is now followed back through every group recorded since the build, down to fragments the snapshot holds: - A group that rewrites fragments into several outputs gives them the same sources, so repeated compactions reach a fragment along exponentially many paths while the history grows linearly. Each fragment is followed once, and each source is returned once. - A source that neither the snapshot holds nor a recorded group produced is a missing link: the fragment is left out, whatever other branches trace back. - Following another covered fragment's sources forward through the group that produced such a fragment no longer marks it fresh again. Those sources vouch for none of the group's other inputs, whose rows the segment cannot translate. The freshness checks along the chain are unchanged, so indexed data that changed along it still leaves the fragment out.
The two saved-segment cases from the lance-format#9769 review, on a stable-row-ID table: a zone map reattached by a raw CreateIndex after cleanup trimmed the record its coverage needs is refused, as invalid when the commit reads the trimmed version and as retryable when it is detached from the version before the trim. A zone map prepared while the record was kept, which cleanup then trims, in one case with its build manifest, is refused as retryable, and committed again from the latest version it covers nothing.
Segments merged after compactions that each rewrite several fragments can index only some inputs of a recorded group: a subset of per-fragment segments, a segment staged over part of a chain's sources, or a saved segment whose coverage lost one output of a group while a sibling is dated by a chain a cleanup broke. In each case the merge claims none of the group's outputs, and lookups match a scan. Tracing finds every such group, and checking freshness can pass its outputs, but neither adds coverage. The history moves coverage across a group only when the segments cover every input, and these tests pin that rule: without it the merges claim outputs holding rows the segments never indexed, and lookups miss them.
…ntal base without lance-format#9947, local only)
…D tables Applies P3's reviewed change (lance-format#9769, as reviewed at 033d7cc and extended at cc4d87b4 with the row-address BTree coverage) onto main and lance-format#9946 without lance-format#9947. Experimental, local only.
…ormat#9947 Without lance-format#9947, nothing validates a saved, prepared or raw segment published after the history it needs was trimmed or cleaned up. The cases that require that validation leave the default suite: - the five tests built entirely on it; - the saved, trimmed case of the republished zone map test; - the cleanup-first order of the racing test; - the stale republication at the end of the address index lifecycle test. The rest of each test stays. They are kept, unchanged, as a diagnostic patch that restores them. On this stack they demonstrate the deferred bug, including wrong answers from a raw replacement.
033d7cc to
6e5dfbb
Compare
wjones127
left a comment
There was a problem hiding this comment.
Thanks for addressing my feedback. One nonblocking question.
| dataset.create_scalar_index("i", index_type) | ||
| dataset.create_scalar_index("i", "ZONEMAP") | ||
| with pytest.raises(ValueError, match="fragment reuse index"): | ||
| dataset.create_scalar_index("category", "NGRAM") |
There was a problem hiding this comment.
question (non-blocking): category already has an NGRAM index (category_idx) from the setup above. Is the ValueError here really the fragment-reuse refusal, or could a duplicate-index error fire first? Consider asserting on a different column or index name so the match="fragment reuse index" can only pass for the intended reason.
…-pk-flush-20261028 # Conflicts: # rust/lance/src/dataset/optimize.rs
A partial compaction relabels every trailing fragment so replacement IDs stay in manifest order. Validation must account for all physical rows moved with a relabeled data and deletion file, not only the live rows.\n\nAdapt the history and compaction fixtures to strict trailing-suffix planning, including concurrent rewrites and retryable conflicts, while retaining stable-row-identity coverage.
…-pk-flush-20261028
There was a problem hiding this comment.
Partial compactions now record metadata-only suffix relabels correctly, including deleted physical offsets. History validation, cleanup and cache identity checks protect recorded mappings across retries and detached commits. Drain pending history before eager BTree remapping.
Late saved, prepared or raw index publication after cleanup can still silently miss matching rows. This inherited, previously accepted limitation remains deferred to #9947; no admission change is requested here. Rebuild against the current dataset instead of reusing saved or prepared metadata across cleanup.
Recording remains limited to debug builds, and costs over long histories remain unmeasured.
Xuanwo
left a comment
There was a problem hiding this comment.
Reviewed the current head, including strict trailing-suffix compaction, metadata-only relabel history recording with deletions, retry/conflict behavior, and the Python stable-row-ID fixtures. Rust history and targeted compaction tests, clippy/format, the full Python optimize suite with recording enabled, and lint pass locally; all applicable CI and the current-head Gatekeeper verdict pass.
## Summary Reverts #8400 and #9769. #8400 makes compaction preserve row order. To do this it relabels every fragment after the first compacted range with fresh IDs, and it makes compaction a retryable conflict with any concurrent `Append` (or `Update` that adds fragments). Under steady ingestion, compaction can then keep retrying and never land. Guaranteeing row order across compaction was previously rejected by the maintainers, so #8400 is rejected as a design. #9769 is built on #8400's metadata-only suffix relabels, so it can't stay without it. It is reverted here and will be redone with its author, without the relabel dependency. ## Merge Risk **Door:** two-way. Neither PR changed the on-disk format. #8400 shipped only in `v15.0.0-beta.4`/`beta.5`; tables it compacted keep valid, ID-sorted manifests and read normally after the revert. **Blast Radius:** concurrency (append/compact become compatible again), compaction, fragment reuse history on stable-row-ID tables. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Record metadata-only suffix relabels with all moved physical row addresses, including deleted rows. Integrate strict trailing-suffix planning with recorded and unrecorded compaction races, and cover the resulting Rust and Python paths. Keep the accepted #9947 follow-up outside this PR scope.
On SRID tables, indexes can rely on FRI to find rows after compaction. We need to protect that history while indexes still need it, and clean it up once they catch up.
This PR checks history chages at commit time so needed mappings can't be removed or corrupted, cleans up old history when indexes no longer need it and prevents readers form using stale cached mappings after the history changes.