Skip to content

fix(mem_wal): read frozen memtables in ShardWriter.lsm_scanner - #9687

Merged
Xuanwo merged 3 commits into
lance-format:mainfrom
zhangyue19921010:fix/python-memwal-scanner-frozen-memtables
Oct 24, 2026
Merged

Xuanwo merged 3 commits into
lance-format:mainfrom
zhangyue19921010:fix/python-memwal-scanner-frozen-memtables

Conversation

@zhangyue19921010

@zhangyue19921010 zhangyue19921010 commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

ShardWriter.lsm_scanner() captured only the active MemTable. A MemTable that has been sealed but not yet flushed is neither the active one nor an SSTable in the manifest, so rows in it were invisible to the scanner until the flush committed. With max_memtable_batches=1 a row written by put was missing from a scan issued right after put returned.

The binding now captures the active MemTable together with every frozen MemTable through in_memory_memtable_refs, the same call the Java binding and the Rust read path use.

This aligns the Python binding with Java. Java's ShardWriter.lsmScanner() already reads both the active MemTable and the frozen MemTables still waiting to be flushed, through in_memory_memtable_refs (java/lance-jni/src/mem_wal.rs). Python only read the active one, so a row that had been sealed but not yet flushed was missing from the scan.

`ShardWriter.lsm_scanner()` captured only the active MemTable. A MemTable
that has been sealed but not yet flushed is neither the active one nor an
SSTable in the manifest, so rows in it were invisible to the scanner until
the flush committed. With `max_memtable_batches=1` a row written by `put`
was missing from a scan issued right after `put` returned.

The binding now captures the active MemTable together with every frozen
MemTable through `in_memory_memtable_refs`, the same call the Java binding
and the Rust read path use.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-python Python bindings bug Something isn't working labels Oct 2, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 2, 2026
@zhangyue19921010 zhangyue19921010 changed the title fix(python): read frozen memtables in ShardWriter.lsm_scanner fix(mem-wal): read frozen memtables in ShardWriter.lsm_scanner Oct 2, 2026
@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 2, 2026
@zhangyue19921010 zhangyue19921010 changed the title fix(mem-wal): read frozen memtables in ShardWriter.lsm_scanner fix(mem_wal): read frozen memtables in ShardWriter.lsm_scanner Oct 2, 2026
@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 2, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 23, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The frozen-aware capture still closes the Python read-your-writes gap after the merge from main. All 23 local MemWAL tests passed, along with rollover checks in both durability modes covering updates, deletes, null values and retained scanners. No blocking issues found.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 23, 2026
@Xuanwo
Xuanwo merged commit d30200e into lance-format:main Oct 24, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants