Skip to content

fix: protect fragment reuse history on stable-row-ID tables - #9769

Merged
Xuanwo merged 21 commits into
lance-format:mainfrom
brendanclement:brendan/fri-srid-history-preservation
Oct 28, 2026
Merged

Xuanwo merged 21 commits into
lance-format:mainfrom
brendanclement:brendan/fri-srid-history-preservation

Conversation

@brendanclement

@brendanclement brendanclement commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

On SRID tables, indexes can rely on FRI to find rows after compaction. We need to protect that history while indexes still need it, and clean it up once they catch up.

This PR checks history chages at commit time so needed mappings can't be removed or corrupted, cleans up old history when indexes no longer need it and prevents readers form using stale cached mappings after the history changes.

@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI A-ci CI / build workflows enhancement New feature or request labels Oct 7, 2026
@brendanclement
brendanclement marked this pull request as ready for review October 8, 2026 02:58

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 8, 2026
merge_existing_index_segments keeps a segment's coverage of a fragment that
compaction produced after the segment was built only if it can trace the
fragment back to the segment's build snapshot through the fragment reuse
index. It looked back one group, so a fragment made by compacting an earlier
compaction's output had a source the snapshot never held: the fragment was
taken as changed and left out of the merged coverage, and committing the
merge over the segment it replaces was then refused for orphaning it.

Each source is now followed back through every group recorded since the
build, down to fragments the snapshot holds:
- A group that rewrites fragments into several outputs gives them the same
  sources, so repeated compactions reach a fragment along exponentially many
  paths while the history grows linearly. Each fragment is followed once,
  and each source is returned once.
- A source that neither the snapshot holds nor a recorded group produced is
  a missing link: the fragment is left out, whatever other branches trace
  back.
- Following another covered fragment's sources forward through the group
  that produced such a fragment no longer marks it fresh again. Those
  sources vouch for none of the group's other inputs, whose rows the segment
  cannot translate.

The freshness checks along the chain are unchanged, so indexed data that
changed along it still leaves the fragment out.
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 13, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Oct 13, 2026
brendanclement added a commit to brendanclement/lance that referenced this pull request Oct 14, 2026
The two saved-segment cases from the lance-format#9769 review, on a stable-row-ID
table: a zone map reattached by a raw CreateIndex after cleanup trimmed
the record its coverage needs is refused, as invalid when the commit reads
the trimmed version and as retryable when it is detached from the version
before the trim. A zone map prepared while the record was kept, which
cleanup then trims, in one case with its build manifest, is refused as
retryable, and committed again from the latest version it covers nothing.
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Oct 14, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Oct 14, 2026
@wjones127
wjones127 self-requested a review October 14, 2026 17:24
Segments merged after compactions that each rewrite several fragments
can index only some inputs of a recorded group: a subset of per-fragment
segments, a segment staged over part of a chain's sources, or a saved
segment whose coverage lost one output of a group while a sibling is
dated by a chain a cleanup broke. In each case the merge claims none of
the group's outputs, and lookups match a scan.

Tracing finds every such group, and checking freshness can pass its
outputs, but neither adds coverage. The history moves coverage across a
group only when the segments cover every input, and these tests pin
that rule: without it the merges claim outputs holding rows the segments
never indexed, and lookups miss them.
…D tables

Applies P3's reviewed change (lance-format#9769, as reviewed at 033d7cc and
extended at cc4d87b4 with the row-address BTree coverage) onto main and
lance-format#9946 without lance-format#9947. Experimental, local only.
…ormat#9947

Without lance-format#9947, nothing validates a saved, prepared or raw segment
published after the history it needs was trimmed or cleaned up. The
cases that require that validation leave the default suite:
- the five tests built entirely on it;
- the saved, trimmed case of the republished zone map test;
- the cleanup-first order of the racing test;
- the stale republication at the end of the address index lifecycle test.

The rest of each test stays.

They are kept, unchanged, as a diagnostic patch that restores them. On
this stack they demonstrate the deferred bug, including wrong answers
from a raw replacement.
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 26, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Oct 26, 2026
@wjones127
wjones127 self-requested a review October 27, 2026 19:05
@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 27, 2026

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing my feedback. One nonblocking question.

dataset.create_scalar_index("i", index_type)
dataset.create_scalar_index("i", "ZONEMAP")
with pytest.raises(ValueError, match="fragment reuse index"):
dataset.create_scalar_index("category", "NGRAM")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question (non-blocking): category already has an NGRAM index (category_idx) from the setup above. Is the ValueError here really the fragment-reuse refusal, or could a duplicate-index error fire first? Consider asserting on a different column or index name so the match="fragment reuse index" can only pass for the intended reason.

…-pk-flush-20261028

# Conflicts:
#	rust/lance/src/dataset/optimize.rs
A partial compaction relabels every trailing fragment so replacement IDs stay in manifest order. Validation must account for all physical rows moved with a relabeled data and deletion file, not only the live rows.\n\nAdapt the history and compaction fixtures to strict trailing-suffix planning, including concurrent rewrites and retryable conflicts, while retaining stable-row-identity coverage.
@lance-gatekeeper lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 28, 2026
@Xuanwo
Xuanwo requested a review from lancedb-robot October 28, 2026 14:26
@lance-gatekeeper
lance-gatekeeper Bot removed the request for review from lancedb-robot October 28, 2026 14:26

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

Partial compactions now record metadata-only suffix relabels correctly, including deleted physical offsets. History validation, cleanup and cache identity checks protect recorded mappings across retries and detached commits. Drain pending history before eager BTree remapping.

Late saved, prepared or raw index publication after cleanup can still silently miss matching rows. This inherited, previously accepted limitation remains deferred to #9947; no admission change is requested here. Rebuild against the current dataset instead of reusing saved or prepared metadata across cleanup.

Recording remains limited to debug builds, and costs over long histories remain unmeasured.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Oct 28, 2026

@Xuanwo Xuanwo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the current head, including strict trailing-suffix compaction, metadata-only relabel history recording with deletions, retry/conflict behavior, and the Python stable-row-ID fixtures. Rust history and targeted compaction tests, clippy/format, the full Python optimize suite with recording enabled, and lint pass locally; all applicable CI and the current-head Gatekeeper verdict pass.

@Xuanwo
Xuanwo merged commit f343c6c into lance-format:main Oct 28, 2026
42 of 44 checks passed
wjones127 added a commit that referenced this pull request Oct 28, 2026
## Summary

Reverts #8400 and #9769.

#8400 makes compaction preserve row order. To do this it relabels every
fragment after the first compacted range with fresh IDs, and it makes
compaction a retryable conflict with any concurrent `Append` (or
`Update` that adds fragments). Under steady ingestion, compaction can
then keep retrying and never land. Guaranteeing row order across
compaction was previously rejected by the maintainers, so #8400 is
rejected as a design.

#9769 is built on #8400's metadata-only suffix relabels, so it can't
stay without it. It is reverted here and will be redone with its author,
without the relabel dependency.

## Merge Risk

**Door:** two-way. Neither PR changed the on-disk format. #8400 shipped
only in `v15.0.0-beta.4`/`beta.5`; tables it compacted keep valid,
ID-sorted manifests and read normally after the revert.
**Blast Radius:** concurrency (append/compact become compatible again),
compaction, fragment reuse history on stable-row-ID tables.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
wjones127 pushed a commit that referenced this pull request Oct 28, 2026
Record metadata-only suffix relabels with all moved physical row addresses, including deleted rows. Integrate strict trailing-suffix planning with recorded and unrecorded compaction races, and cover the resulting Rust and Python paths. Keep the accepted #9947 follow-up outside this PR scope.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-ci CI / build workflows A-java Java bindings + JNI A-python Python bindings bug Something isn't working enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants