Repository navigation
Conversation
`ShardWriter.lsm_scanner()` captured only the active MemTable. A MemTable that has been sealed but not yet flushed is neither the active one nor an SSTable in the manifest, so rows in it were invisible to the scanner until the flush committed. With `max_memtable_batches=1` a row written by `put` was missing from a scan issued right after `put` returned. The binding now captures the active MemTable together with every frozen MemTable through `in_memory_memtable_refs`, the same call the Java binding and the Rust read path use.
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The frozen-aware capture still closes the Python read-your-writes gap after the merge from main. All 23 local MemWAL tests passed, along with rollover checks in both durability modes covering updates, deletes, null values and retained scanners. No blocking issues found.
Xuanwo
approved these changes
Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ShardWriter.lsm_scanner()captured only the active MemTable. A MemTable that has been sealed but not yet flushed is neither the active one nor an SSTable in the manifest, so rows in it were invisible to the scanner until the flush committed. Withmax_memtable_batches=1a row written byputwas missing from a scan issued right afterputreturned.The binding now captures the active MemTable together with every frozen MemTable through
in_memory_memtable_refs, the same call the Java binding and the Rust read path use.This aligns the Python binding with Java. Java's
ShardWriter.lsmScanner()already reads both the active MemTable and the frozen MemTables still waiting to be flushed, throughin_memory_memtable_refs(java/lance-jni/src/mem_wal.rs). Python only read the active one, so a row that had been sealed but not yet flushed was missing from the scan.