Skip to content

fix(index): apply segment ownership in shared-scan batch IVF search - #10

Closed
sezruby wants to merge 563 commits into
base/adaptive-nprobes-batchfrom
fix/batch-ivf-segment-ownership
Closed

sezruby wants to merge 563 commits into
base/adaptive-nprobes-batchfrom
fix/batch-ivf-segment-ownership

Conversation

@sezruby

@sezruby sezruby commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Published upstream as lance-format#9638. This fork PR tracks the same branch.

Summary

Fix an existing stale-vector correctness gap in shared-scan batch IVF search, following up lance-format#7640.

An in-place vector update (update_columns / Operation::Update) can preserve a fragment's ID and row IDs while pruning that fragment from the old index segment's fragment_bitmap. After optimize_indices(append) indexes the replacement in another delta, the old index file still physically contains the pre-update vector.

ANNIvfBatchExec searched all deltas with the shared dataset prefilter, so the old segment could return that stale vector. Single-query search already handles segment ownership; this patch reuses its existing helpers:

  • Search each delta with prefilter_for_segment, restricting candidates before local top-k.
  • Apply restrict_to_segment before the cross-delta merge, with the same all-segments-have-bitmaps gate as single-query search.
  • Run prefilter_for_partition once per distinct loaded batch partition, before CPU scoring. Fully owned partitions regain NoFilter and bulk scoring; partitions containing stale entries keep ownership filtering before top-k. This preserves the single-query safeguard from fix(index): filter stale vector segment rows before top-k lance-format/lance#8351, bounded streaming, and load/scoring overlap.

No public API, output-schema, or index-format changes. Adaptive probing is a separate follow-up.

Regression coverage

test_batch_knn_skips_rows_owned_by_newer_delta covers both physical row addresses and stable row IDs:

  • A query at the stale vector must not return the updated row as a nearest neighbor.
  • A query at the fresh vector returns the updated row once.
  • Both queries return k rows, with distances matching single-query results.

Removing the ownership filtering reproduces the failure in both row-ID cases.

The append-only fast-path test now exercises actual batch partition search with a segment prefilter. A FLAT / 4-bit PQ / 8-bit PQ matrix additionally verifies:

  • Coverage is checked once per distinct partition, not once per query.
  • Fully owned partitions perform no per-row mask reads.
  • Mixed ownership excludes the closest stale candidate before heap truncation and still fills top-k.
  • Empty ownership returns no candidates; ordinary user filters remain active.
  • Neighbors/distances match an exhaustive same-index oracle, and recall against brute-force search is at least 0.5.

All three new matrix cases failed on the previous PR head because batch search never checked partition coverage; they pass with the shortcut restored. These are deterministic scoring-path assertions, not new timing measurements.

Validation

Validated on upstream base f4f724131.

  • cargo test -p lance --lib: 4,509 passed, 0 failed, 3 ignored.
  • Batch tests with one compute worker and streaming batch size 1: 178 passed; the append-only regression also passed separately.
  • Workspace clippy (--all --tests --benches -- -D warnings) and lance all-features clippy: clean.
  • cargo fmt --all -- --check: clean.

LuciferYang and others added 28 commits October 23, 2026 13:19
…ce-format#9930)

`RepDefUnraveler::decimate` ended in `todo!("Not yet supported
FSL<...List<...>>")` whenever the fixed-size-list layer was reached with
repetition levels present. A `FixedSizeList<Struct<List<Int32>>>` column
writes fine at the stable file version, and every read of it then
panicked inside a decode task, so the caller got `Wrapped error: task 23
panicked with message "not yet implemented: Not yet supported
FSL<...List<...>>"` rather than a typed error.

It now returns `Error::not_supported`, which `rust/AGENTS.md` asks for
on unimplemented paths. The zero-dimension arm of the same function
already returns a typed error; lance-format#8618 made that one `invalid_input`,
which fits there because a zero dimension is bad caller data rather than
a missing feature. Reads of such a column still fail, but with a
descriptive error instead of an unwound panic.

lance-format#9926 has the per-version table. V2_1 rejects the schema at write time,
so the versions that accept it have no write-side gate. Whether a given
column hits this also depends on which struct child carries the list,
because `RepDefStructDecodeTask` keeps child 0's repdef and discards the
dense siblings. That is why
`test_fsl_struct_random::case_3_struct_with_list` passes today: its
fixture puts `id` first.

Fixes lance-format#9926.

## How was this patch tested

`test_repdef_fsl_over_list_is_not_supported` builds the repdef layers
the writer emits for `FixedSizeList<Struct<List<Int32>>>`
(fixed-size-list, struct, list offsets, leaf validity), consumes the
inner layers in decode order, and asserts the fixed-size-list layer
returns `NotSupported`. Restoring the `todo!()` makes it fail on the
panic instead.

A scratch probe over all four selectable file versions, not part of the
patch, moved the V2_2 and V2_3 reads from the wrapped panic message to
`Not supported: Reading a column that combines a fixed-size-list with a
list-like layer ...`. V2_0 and V2_1 still reject the write.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
`modifies_same_metadata` compared `schema_metadata_updates` with a
blanket `is_some() && is_some()` check, while the table-metadata
comparison a few lines above routes through `update_maps_conflict`,
which conflicts only on a shared key or when either side sets `replace`.
`UpdateMetadataBuilder` produces merge-mode maps by default, so two
concurrent commits that each set a different schema-metadata key were
rejected with a non-retryable `IncompatibleTransaction`. lance-format#9972 has the
repro and the error the second writer gets.

Route schema metadata through `update_maps_conflict` as well. Either
side setting `replace` still conflicts, which keeps the behavior the
whole-write path depends on.

Fixes lance-format#9972.

## How was this patch tested

`test_schema_metadata_conflict_is_per_key` mirrors the neighbouring
`test_table_metadata_conflicts_on_same_key`: the same key conflicts,
different keys do not, and a replace-mode map conflicts. Restoring the
blanket check fails the disjoint-key assertion.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
…ion (lance-format#9976)

`is_field_binding_fully_rewritten` checked materialization with a plain
`fields.contains(&field_id)`, while the sibling
`file_materializes_field`, used for the same concept elsewhere in this
validation, recurses into a struct's children. STABLE files record only
leaf ids, so a `Merge` that changes a struct's own binding while
rewriting every fragment was rejected, and the prior-file-replaced check
below it was vacuous for a struct id. lance-format#9973 has the error text and the
public commit surface that reaches it.

Both sides now go through `file_materializes_field`, with the field each
side is actually about: a new file has to carry data for the proposed
binding, and the files that have to be gone are the ones carrying the
prior binding. Passing both fields in also drops the schema lookup the
first version used, which searched only the top-level fields and left a
nested struct rejected.

Fixes lance-format#9973.

## How was this patch tested

`test_merge_struct_binding_fully_rewritten_stable_files` has two cases,
a top-level struct and a nested one. Each flips the nullability of the
struct whose binding changes and rewrites the single fragment with a
V2_2 file that lists only leaf ids. Both fail against main: the
top-level case on the plain `contains`, the nested case on a lookup that
only sees top-level fields.

`test_merge_struct_binding_checks_the_proposed_field` covers the other
direction: the struct keeps its id and flips to nullable while its child
becomes a different field, and the replacement file carries only the
prior child. Matching new files against the prior field accepts that
merge even though the file holds no data for the proposed binding;
matching against the proposed field rejects it.

---------

Co-authored-by: Xuanwo <github@xuanwo.io>
…ance-format#10211)

fixed lance-format#10210
schema_fragments_modern_valid only checked that each data file had at
least one field, so committing an Append whose fragments omit a column
declared non-nullable succeeded. That column is then physically absent
and read back as all-null, violating the schema -- silent since lance-format#9562
(previously an Arrow non-null panic on read). It is reachable through
the low-level write_fragments + LanceOperation.Append + commit path,
which never runs check_compatible.

Require every non-nullable field whose ancestors are all non-nullable to
be present in each fragment's data files, mirroring the v1
schema_fragments_legacy_valid check scoped to non-nullable fields.
Nullable fields (and anything under a nullable ancestor) may still be
omitted per the subschema-insert contract (lance-format#3016 / lance-format#3041).

Change-Id: I5a9a018272ac0b5f5941c2d6ff3e705bdcf647f7

---------

Co-authored-by: wangpeipei05 <wangpeipei05@baidu.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
…at#10080)

Closes lance-format#10078

## Motivation

pylance builds with the `geo` feature, which is a default feature of the
`lance` crate, so its spatial predicates run through geodatafusion. When
the constant geometry is the first argument, geodatafusion's
`relate_impl` computes the DE-9IM matrix for (B, A) but reads it as (A,
B). As a result, `ST_Contains(<region>, geom)` returns false for the
points inside the region. `delete("NOT ST_Contains(<region>, geom)")`
deletes every row, including the ones inside the region. A NULL constant
geometry panics the scan. lancedb/lancedb#4555 is about to turn on `geo`
in the LanceDB Python and TypeScript SDKs, so more users will hit this.

The bug is in geodatafusion 0.5.0, which `main` pins, and in 0.6.0,
which lance-format#9979 moves to. The upstream fix,
datafusion-contrib/geodatafusion#111, is still open. This PR fixes the
bug in Lance, so it works with either version and doesn't wait for a
geodatafusion release.

## What changes

`lance-geo` now registers its own versions of the ten relate predicates:
`st_contains`, `st_within`, `st_covers`, `st_coveredby`, `st_crosses`,
`st_disjoint`, `st_equals`, `st_intersects`, `st_overlaps` and
`st_touches`. These replace geodatafusion's registrations of the same
names. They live in `rust/lance-geo/src/udf.rs`. That file was already
on `main`, but no `mod udf` declaration compiled it, and
`geodatafusion::register` already registers everything it listed, so
this PR replaces its contents. Each predicate wraps geodatafusion's
implementation and changes three things:

- **Scalar first, array second.** The call is evaluated as the converse
predicate with the arguments swapped: `ST_Contains(c, x)` =
`ST_Within(x, c)` and `ST_Covers(c, x)` = `ST_CoveredBy(x, c)`. The
symmetric predicates are their own converse. This uses geodatafusion's
(array, scalar) path, which is correct and keeps its prepared-geometry
fast path.
- **NULL geometry.** A NULL scalar geometry in either position returns
NULL instead of panicking.
- **Index use.** `ScalarUDFImpl::simplify` rewrites `f(<literal>, x)` to
`converse(f)(x, <literal>)`. The R-tree matcher (`visit_scalar_fn`) only
takes the indexed column from the first argument, so constant-first
predicates never used the index before this change; now they do. The
rewrite also covers second arguments that are not columns, such as
`ST_Contains(<region>, ST_Point(x, y))`.

## Design

The fix is at the UDF level rather than in the planner, because that is
the one place every evaluation path shares. All of these paths resolve
the functions by name from a registry that
`lance_geo::register_functions` fills:

- the scanner filter
- `delete`, `update` and `merge_insert` conditions
- Substrait filters
- `Dataset::sql`
- downstream sessions built with
`lance_datafusion::udf::register_functions` or `new_session_context`,
which is how LanceDB sets up its sessions

The argument-order rewrite is in the UDF's `simplify` hook rather than
in a Lance-specific planner pass. DataFusion's `ExprSimplifier` runs
that hook both for `Planner::optimize_expr` and for the DataFusion
optimizer behind `Dataset::sql`. Constant folding runs first, so
`ST_GeomFromText('...')` has already become a literal when the rewrite
sees it. The invoke-time swap is still needed for expressions that reach
execution without being simplified, such as a raw `Expr` passed to
`Scanner::filter_expr`.

There are no `Cargo.toml` or `Cargo.lock` changes.

## Affected predicates (verified)

I compared all ten relate predicates, in both argument orders, against
`geo::Relate`. The test geometries were points, lines and polygons that
lie inside a square, on its boundary, or outside it, or that overlap,
cross, equal, touch or enclose it: 156 pairs per predicate. This used a
temporary `geo` dev-dependency that is not part of this PR.

| Predicate | geodatafusion 0.5.0, scalar first | With this PR |
| --- | --- | --- |
| `ST_Contains` | 30 of 156 pairs wrong | all correct |
| `ST_Within` | 30 of 156 pairs wrong | all correct |
| `ST_Covers` | 44 of 156 pairs wrong | all correct |
| `ST_CoveredBy` | 44 of 156 pairs wrong | all correct |
| `ST_Crosses`, `ST_Overlaps` (including mixed dimensions),
`ST_Touches`, `ST_Equals`, `ST_Intersects`, `ST_Disjoint` | all correct
| all correct |

Every (array, scalar) result was already correct before this change. The
NULL panic affects all ten predicates, with the NULL in either position.

## Testing

`lance-geo` unit tests, run with `--features geo`:

- Every predicate, with the constant in either position, against
geodatafusion's (array, array) evaluation. That path relates each pair
of rows with `geo::Relate` in argument order.
- The constant-first result against the column-first converse.
- Known answers for the asymmetric predicates.
- NULL constants, typed and untyped, in both positions, on both a
non-empty and an empty batch.
- The `simplify` rewrite.
- That `register_functions` installs the wrappers.

`lance` dataset tests in `dataset_geo.rs`. Each runs with and without an
R-tree, on a dataset with several fragments that includes a NULL
geometry:

- **Reads**, through the scanner filter, `Dataset::sql`, and a
`SessionContext` set up with `register_functions` and
`LanceTableProvider`. The constant-first result must equal the
column-first converse and the expected rows. When the column is indexed,
the plan for the constant-first form must show `ScalarIndexQuery`.
- **Wrong-row mutation regressions.** `delete` with `NOT ST_Contains`,
`ST_Contains`, `NOT ST_Covers`, `NOT ST_Within` and `NOT ST_CoveredBy`,
and `update` with `ST_Contains`, `NOT ST_Contains`, `ST_Covers` and `NOT
ST_Within`. Each test asserts exactly which rows survive or change.
- **`merge_insert`** with
`WhenNotMatchedBySource::delete_if("ST_Contains(<region>, geom)")`.
- **NULL constant geometry**, in reads and in `delete`.

Python tests in `test_geo.py`: constant-first reads, `update` and
`delete`, each with and without an R-tree, plus the NULL constant.

All of the new Rust and Python tests fail when the wrappers are not
registered, and pass when they are. With only the `simplify` rewrite
disabled, results stay correct but the index-use assertions fail.

Commands run:

- `cargo fmt --all -- --check`
- `cargo clippy --profile ci --locked -p lance-geo --features geo
--all-targets -- -D warnings`
- `cargo clippy --profile ci --locked --all --tests --benches -- -D
warnings`
- `cargo test --profile ci -p lance-geo --features geo`
- `cargo test --profile ci -p lance --lib -- geo rtree st_intersects`
- `cargo test --profile ci -p lance-index --features geo --lib -- rtree
geo`
- From `python/`, with the `geo` extra installed: `uv run make lint` and
`uv run pytest python/tests/test_geo.py`. The new tests pass.
`test_staged_rtree_after_rewrite_columns` fails with the same assertion
on a build without this change.

CI installs only the `tests` and `ray` extras, so it skips `test_geo.py`
(`importorskip("geoarrow.rust.core")`). The Rust tests cover the same
behavior.

## Follow-up

Once Lance pins a geodatafusion release that includes
datafusion-contrib/geodatafusion#111, remove the argument swap and NULL
handling in `RelateUdf::invoke_with_args` in `udf.rs`. The `simplify`
rewrite does not depend on the upstream bug and keeps constant-first
predicates indexable, so it should stay.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
…fix (lance-format#10228)

`extract_like_leading_prefix` now reads a `LIKE` pattern the way the
executor does. Fixes lance-format#10226.

The extractor makes a single left-to-right pass. An escape consumes
whatever character follows it, as arrow's LIKE kernel does, so `a\b%`
gives the prefix `ab` instead of `a\b`, and the wildcard check uses the
same pass, so `\\\%` is the literal `\%` rather than a prefix on it.
Before, a BTREE answered `LIKE 'a\b%'` with rows starting with `a\b` and
no recheck, and a multi-fragment ZONEMAP pruned the fragments holding
the matching rows. A non-backslash `ESCAPE` character now gets no index
acceleration: DataFusion's physical planner refuses it, and an index
query that replaces the predicate would answer it instead. So `LIKE
'a!%%' ESCAPE '!'` on a BTREE-indexed column, which used to return rows,
now fails the way it already did without an index.

`starts_with` is unaffected: DataFusion rewrites it into a pattern that
only escapes `\`, `%` and `_`, which both versions read the same way.

## How was this patch tested

`test_extract_like_leading_prefix` now expects `foo\bar%` to give
`foobar`, where it asserted `foo\bar` before, and adds `\\\%`, `\\\%%`
and an `ESCAPE '!'` case. `test_like_expression_parsing` plans `color
LIKE 'a\b%'` as `LikePrefix("ab")` with no refine and leaves `ESCAPE
'!'` unaccelerated. `test_like_prefix_escapes_match_unindexed_scan` runs
`LIKE 'a\b%'` and `LIKE '\\\%'` through a BTREE and a three-fragment
ZONEMAP and compares the rows with an unindexed scan; both cases fail
without the change.
…ance-format#10175)

## Problem

A merge insert that deletes matched rows, inserts nothing, and deletes
target rows not matched by the source (`when_not_matched_by_source` =
`Delete` or `DeleteIf`) fails with an internal error whenever the source
carries a dataset column other than the key:

```
Dataset field "k" missing from merge insert input schema — this indicates a logical plan pruning bug
```

Example: target `(id, k, emb)`, source `(id, k)`, `on = "id"`. The same
merge with a source of only `id` works.

Cause: two checks disagree on what a delete-only merge is.
- `MergeInsertWriteNode::necessary_children_exprs` pruned the source to
its key columns for any `Delete` without inserts.
- `MergeInsertPlanner` only picks `DeleteOnlyMergeInsertExec` when
`when_not_matched_by_source` is also `Keep`.
- Otherwise it picks `FullSchemaMergeInsertExec`. That exec looks up
every dataset column by name and finds `k` pruned away.

## Fix

In `necessary_children_exprs` (`merge_insert/logical_plan.rs`):

- Prune the source to its key columns only when
`MergeInsertPlanner::is_delete_only` holds, the same check that picks
the exec.
- Otherwise keep only the source columns that exist in the dataset (plus
the source sentinel), not every source column.
  - No exec reads a source column the dataset lacks.
- Such columns can only reach this plan in merges that write nothing,
because updates and inserts require the source to be the full dataset
schema or a subset of it.
- Dropping them keeps the earlier pruning for a source like `(id,
extra)` and stops `extra` from riding through the join.

## Effect

Benchmark setup:
- Target: 1M rows, 6 columns, including a 32-d vector and a struct.
- Source: 510K rows.
- Release build, one merge per process, main and this PR interleaved,
medi

| Case | main | this PR |
|---|---|---|
| Matched `Delete` + not-matched-by-source `Delete`/`DeleteIf`, source
car| error | works; deletes the same rows as a key-only source |
| Matched `DoNothing` + not-matched-by-source `DeleteIf`, source carries
a column not in the dataset | 627 ms, +358 MB peak RSS | 409 ms, +162 MB
peak RSS |
| 9 other combinations (delete-only, upsert, delete + insert, ...) | — |
identical physical plans and results; timings within noise |
…ns (lance-format#10135)

A projection expression that uses a system column inside a larger
expression now loads that column. Fixes lance-format#10129.

`ProjectionBuilder::add_column` set the flags that request `_rowid`,
`_rowaddr`, `_rowoffset`, `_row_last_updated_at_version` and
`_row_created_at_version` only when the whole expression was that
column, so `columns={"next": "_rowid + 1"}` failed with `No field named
_rowid`. The check now runs over every column the expression references,
which also covers the bare-column case it replaces. This applies to
`Scanner::project_with_transform` and to `take` with SQL projections.

## How was this patch tested

`test_scan_with_wildcard` projects each of the five system columns plus
one and checks the output schema, and checks that `_rowoffset + 1`
yields 1 to 100. It fails without the change.
…lance-format#10134)

`is_substrait_compatible` now recurses into `LargeList` and `Map` the
way it already does into `List`. Fixes lance-format#10128.

DataFusion's producer encodes the whole schema, not only the columns a
filter references, and has no Substrait type for `FixedSizeList`.
`prune_schema_for_substrait` drops such columns before encoding, but a
`large_list<fixed_size_list>` or `map<_, fixed_size_list>` column was
reported compatible and kept, so encoding any filter on that dataset
failed with `Unsupported cast type: ... FixedSizeList`. That is the
encoding `filtered_read_exec_to_proto` does for a filtered read. The
decoder takes the pruned schema from the message, so it needs no change.

This complements lance-format#9057, which checks the columns a filter references
with `is_substrait_compatible` before pruning; with this change that
check also catches a filter on a `large_list<fixed_size_list>` column.

## How was this patch tested

`test_prune_schema_for_substrait_nested_list_types` builds a schema with
a plain large list, a plain map, and a large list and a map holding
fixed-size lists. It asserts that only the two holding fixed-size lists
are dropped and round-trips a filter on another column through the
pruned schema. It fails without the change.
…ormat#10133)

`contains_tokens(text, query)` read only the first row of `query` and
applied it to every row. Fixes lance-format#10127.

The UDF now passes `make_scalar_function` hints so a literal query stays
one value, tokenized once per batch, and a column query is read row by
row. A NULL text or NULL query gives NULL. Before, a NULL query read as
`""`, an empty token list that every text contains, so
`contains_tokens(text, NULL)` matched every non-null row; a column query
applied its first row's value to all rows.

The FTS index only takes a non-null string literal, so both cases ran
through the UDF with or without an index. Results for a non-null literal
query are unchanged.

## How was this patch tested

`test_contains_tokens_null_query_matches_nothing` passes a NULL literal,
and `test_contains_tokens_reads_a_column_query_per_row` passes a column
whose first value differs from the second, so reading only the first row
gets the second row wrong. `test_sql_contains_tokens` adds
`contains_tokens(text, NULL)` through SQL, before any index is built and
with a suitable FTS index, and asserts no rows. All three fail without
the change.
…sts, narrow under NOT (lance-format#10131)

Three fixes to how `lance-datafusion` turns filter and expression
strings into DataFusion expressions. Fixes lance-format#10125.

`Planner::value` returns an `InvalidInput` error for a SQL value it has
no mapping for, instead of reaching `todo!()`. Placeholders (`:id`,
`$1`, `?`) and national strings (`N'abc'`) used to panic, including
inside an array literal; from Python that panic is a `PanicException`,
which `except Exception` does not catch. `is_valid()` now requires
exactly one argument: with none it panicked on `args[0]`, and extra
arguments were dropped.

`parse_sql_expr` rejects input that parses to more than one `SELECT`
item. It used to keep the first item and drop the rest, so `update({"x":
"coalesce(x), 0"})` committed `x = coalesce(x)`. Every caller of
`Planner::parse_expr` (`update`, `add_columns`, `columns={...}`
projections) passes one expression per string.

`resolve_expr` recurses into `NOT` and `IS [NOT] TRUE/FALSE`, so the
literal inside is narrowed to the column type the same way it is
outside. Without that, on a `Float32` column `NOT (f = 0.1)` compares
`CAST(f AS Float64)` with `0.1`, and the row holding `0.1f32` matches
both `f = 0.1` and `NOT (f = 0.1)`. One consequence is that a literal
`resolve_expr` already refuses at the top level is now refused under
these wrappers as well: `NOT (i = 1.5)` on an `Int32` column used to
plan through DataFusion's own coercion and now returns the same error as
`i = 1.5`.

## How was this patch tested

`test_parse_filter_rejects_unsupported_literals` covers each placeholder
form, a national string and one inside an array literal, and asserts the
variant and message. `test_parse_filter_checks_is_valid_arity` covers
zero and two arguments. `test_parse_expr_rejects_multiple_expressions`
uses the misplaced-parenthesis input, and `test_update_validation`
passes the same input through `UpdateBuilder::set`.
`test_parse_filter_narrows_under_not_and_is` evaluates `=`, `>`,
`BETWEEN` and `IN` on a `Float32` column and asserts that `NOT (...)`,
`IS FALSE` and `IS NOT TRUE` return the complement, and `IS TRUE` and
`IS NOT FALSE` the same rows. Each of these fails without the change.
…s its field (lance-format#10123)

Closes lance-format#9724.

A `DataOverlay` staged at version N still committed after a concurrent
`Project` at N+1 dropped the field it overlays. `check_data_overlay_txn`
treated every `Project` as compatible, and the overlay build arm doesn't
look at the schema, so the overlay landed covering only fields that no
longer exist. lance-format#9314 handles the other order, where `Project` prunes
overlays that are already committed.

This makes `Project` a retryable conflict for an overlay when the
projected schema no longer contains one of the overlay's fields, which
is the first option the issue gives. A projection that keeps all of the
overlay's fields (a rename, or dropping some other column) is still
compatible.

Testing: `test_data_overlay_conflicts` gained two `Project` cases, one
keeping the overlaid field (compatible) and one dropping it (retryable).
The dropping case fails on main. `cargo test -p lance --lib
io::commit::conflict_resolver` and `cargo test -p lance --lib overlay`
pass, `cargo fmt --all` is clean, and `cargo clippy -p lance --lib
--tests -- -D warnings` passes; I didn't run the full-workspace clippy.

lance-format#9220 leaves the `(DataOverlay, Project)` cell out of its conflict
matrix because of this bug; with this change it should be able to go
back in.

I used an AI coding assistant to help write this change and the test; I
reviewed the diff and ran the checks above.
…e-format#10161)

## Problem

Every node of a MemTable's HNSW graph keeps, per level, a published
neighbor list behind an `ArcSwap`. Searchers read level 0 from the
graph's packed level-0 snapshot (`packed_level0`), which `publish_dirty`
rebuilds before any new row becomes visible, so a node's own published
level-0 list is never read for a node a search can reach.

It is still expensive to free. Dropping an `ArcSwap` walks arc-swap's
per-thread debt list, so freeing a graph drops one per node per level,
and the graph has a node for every id up to its capacity. A stack sample
of a server process freeing a ~414k-row graph at a 1M capacity shows
`drop_glue::<HnswGraph>` → `drop_glue::<Vec<LevelLinks>>` →
`arc_swap::debt::Debt::pay_all` for about 7 s of CPU. The
frozen-MemTable grace sweep frees an evicted generation while holding
the shard writer lock, so puts to that shard wait for the whole free.

## Fix

- `LevelLinks::published` is `None` on level 0. Upper levels, which
about one node in `m` reaches, still publish per node.
- `LevelLinks::linked_bytes` no longer charges a per-node level-0 list;
level 0 is charged in the packed snapshot.
- `mem_wal_hnsw_bench` prints the graph's resident bytes and drop time
(`lifecycle` line).

## Measurements

`mem_wal_hnsw_bench`, `release-with-debug`, dim 64, m 12,
ef_construction 64, 1,000-row batches, 18 threads, Apple M-series; mean
of two runs.

| rows | capacity | drop, before → after | resident MB, before → after |
|---|---|---|---|
| 100k | 100k | 0.13 s → 0.024 s | 55.5 → 41.9 |
| 100k | 1M | 1.11 s → 0.13 s | 142.6 → 136.9 |
| 414k | 1M | 1.17 s → 0.14 s | 286.2 → 235.3 |

Recall@10 is unchanged within run-to-run variance.

`Option<ArcSwap<_>>` adds 8 bytes to every level header (`ArcSwap` has
no niche), about 8.7 MB per million reserved nodes on a graph with no
rows; each inserted node saves its level-0 list and `Arc` block (about
128 bytes at m 12), so only very sparsely filled graphs grow. lance-format#10162
allocates slots only for rows that arrive, which removes the fixed part.
The grace sweep still frees an evicted generation under the shard writer
lock; this change makes that free cheap rather than moving it.

## Tests

- `test_parallel_insert_searches_visible_graph` now also checks that no
node keeps a level-0 `ArcSwap` and that upper levels still do; it fails
with the per-node level-0 list restored.
- `test_lance_hnsw_batch_edges_stay_inside_the_exported_prefix`, which
reads the graph while a writer inserts, now also checks, in the order a
search reads, that every visible node is in the packed level-0 snapshot.
It fails if the snapshot is not rebuilt, and if the visible length is
published before the snapshot.
- `cargo nextest run -p lance --lib -E 'test(/dataset::mem_wal/) |
test(/hnsw/)'`: 1052 passed.
- `cargo fmt --all` and `cargo clippy -p lance --tests --benches -- -D
warnings`: clean.

lance-format#10162 changes the same file (node slots grow with rows); whichever
merges second needs a one-line update to that test's node iteration.
…ormat#9964)

Hierarchical k-means trained each node in `grow_proportional_subtree` on
a prefix of its rows, assuming the input was already shuffled.

In the non-nullable vector sampling path,
[`generate_random_indices`](https://github.com/lance-format/lance/blob/e483590d6df5b767e190913eeb35aa0592614a0a/rust/lance/src/index/vector/utils.rs#L1099-L1119)
selects random row positions and then sorts them with
`indices.sort_unstable()`.
[`sample_fsl_uniform`](https://github.com/lance-format/lance/blob/e483590d6df5b767e190913eeb35aa0592614a0a/rust/lance/src/index/vector/utils.rs#L879-L935)
reads those positions through `dataset.take()` in chunks, preserves
chunk order with `.buffered(batch_readahead)`, and concatenates the
vectors without shuffling them. The full sample is uniformly selected,
but its order follows source-row positions; taking a prefix of it is
therefore not uniform subsampling.

When similar vectors are ingested together—for example, batches from the
same source or category—the prefix overrepresents early batches. This
biases the hierarchy's splits and centroids, which can worsen IVF
partition skew and contribute to empty partitions.

Subsample uniformly without replacement at each node, using the existing
row-index buffer and node-derived seed. All node rows still participate
in assignment, centroid quotas, and leaf means. The upstream sorted
reads are preserved. Add regressions for ordered groups and rows
appended when merging zero-quota children.

Validation:
- `cargo test --profile ci --locked -p lance-index
vector::kmeans::tests` (38 passed; one existing benchmark ignored)
- `cargo fmt --all`
- `cargo clippy --profile ci --locked --all --tests --benches -- -D
warnings`
- `git diff --check`
…ance-format#9963)

Packed-struct encoding accepts two child kinds it cannot encode, and
both fail destructively. lance-format#9962 has the details and the measured
per-version behavior. The short version: an all-null or
dictionary-encoded child panics inside `DataBlock::from_arrays`, and a
Boolean child writes a column whose values were never stored, or panics
the writer once a page reaches 4096 rows.

Two changes, one per hole.

`StructDataBlock::compute_stat` summed `MaxLength` over every child with
`expect_single_stat`, which panics for a child that carries no such
statistic. It now sums into an `Option<u64>` and leaves the statistic
out when any child has no width to contribute, because an unknown width
is not a zero width. `FixedWidthDataBlock::compute_stat` always inserts
`MaxLength`, so the sum is absent exactly when some child is not a
fixed-width block, which is the case `try_fixed_packed_struct_miniblock`
already refuses with a typed error. The only reader of a struct block's
`MaxLength` is `is_narrow`, which already treats it as optional and
falls back to full-zip.

`try_fixed_packed_struct_miniblock` now also rejects a child whose
`bits_per_value` is not a positive multiple of 8.
`has_variable_width_child` only looks at the block kind, so a Boolean
child of one bit passed it, and the packing then wrote `bits_per_value /
8 = 0` bytes for that child while counting its bit in the declared row
width. The full-zip packing already refuses a child that is not
byte-aligned and the decoder requires a positive width, so this is the
mini-block path catching up.

Fixes lance-format#9962.

## How was this patch tested

`test_struct_stat_without_child_max_length` builds a struct with one
all-null child, asserts the block keeps `DataSize` but has no
`MaxLength`, and asserts `try_fixed_packed_struct_miniblock` returns a
typed error for it. Restoring the `expect_single_stat` sum makes it
panic instead.

`fixed_packed_struct_miniblock_rejects_sub_byte_child` asserts the typed
rejection for a packed `{Boolean, Int32}` struct, naming the offending
width. Without the new check the function returns an encoder and the
test fails. The decoder side of the same contract is already pinned by a
case in `mini_block_descriptor_errors_are_owned_by_codecs`.

End to end, with a scratch probe over V2_1, V2_2 and V2_3, a packed
`{Boolean, Int32}` column went from "write succeeds, read fails" at 3
rows and a writer panic at 4096 rows to the same typed error in all six
cases. A packed struct with an all-null or dictionary child went from a
panic out of `write_batch` to `Not supported: Per-value compression not
yet supported for block type AllNull` on V2_2 and V2_3, `Not supported:
Variable packed struct encoding is not enabled by the selected file
format` on V2_1, and `Invalid user input: Packed struct encoder
currently only implemented for fixed-width data blocks` on V2_0.
Closes lance-format#9955

Hi team, in the current implementation, it reads to me, even when a
`Binary` or `Utf8` field explicitly requests lz4, Lance's variable-width
per-value encoding path can use Zstd instead.

Simply honoring `lz4` is not enough: full-zip stores no payload for null
values, and the LZ4 reader [fails on that empty
payload](https://github.com/lance-format/lance/blob/0d42b38f4c8a8425fe479cec06676ad93d781b7b/rust/lance-encoding/src/encodings/physical/block.rs#L411-L415).
Released readers (including v13) have the same check, so stable-format
files with nullable LZ4 full-zip pages would be unreadable by them.

## What this PR does

- **Reader**: `CompressedBufferEncoder::per_value_decompress` now skips
empty slots instead of passing them to the decompressor. This is safe
for every codec: LZ4 and Zstd length-prefix every valid value (including
empty strings), so an empty slot can only be a null.
- **Writer**: `try_variable_width_per_value` takes a new
`PerValueGeneralCompression` argument that controls which explicitly
requested codecs the per-value path honors.
- Stable 2.1 / 2.2 pass `ZstdOnly`, which keeps the existing selection:
only an explicit `zstd` is honored, and other requests fall back to the
default compressor. Stable output is unchanged, so released readers keep
reading it.
- Unstable 2.3 passes `Any`, which honors the requested codec, including
`lz4`.
- `fsst` keeps the existing default fallback in both modes, since FSST
cannot be used as a general buffer compressor.

## Compatibility

No change to stable-format output. Only the unstable 2.3 format can now
emit LZ4 per-value pages with nulls, and those require a reader with
this fix. Once 2.3 is stabilized, its minimum reader will already
include the fix, so LZ4 can reach default users without breaking
released readers.

For stable formats I kept the existing fallback rather than rejecting
the `lz4` selection, because rejecting would turn writes that succeed
today into errors for users who already set `lz4` on these columns.
…eto the first (lance-format#9895)

## Summary

A vector column can carry more than one index (e.g. an L2 index and a
cosine index). When a nearest-neighbour query specified a `metric_type`,
the scanner picked the **first** index keyed on that column and then
used the query metric only to **veto** it. On a mismatch it warned and
brute-forced the whole column — even when a second index on the same
column had exactly the requested metric.

This makes the metric part of index **selection** instead of a post-hoc
veto, so a cosine query on a column holding both `idx_l2` and `idx_cos`
now uses `idx_cos`.

Fixes lance-format#9873.

## Root cause

In `create_plan`'s vector-index branch
(`rust/lance/src/dataset/scanner.rs`), the candidate was chosen with

```rust
indices.iter().find(|i| i.fields.first() == Some(&column_id) && index_type_is_known(i))
```

The predicate filters on the keyed column and index type but **not** the
metric. The query's `metric_type` was applied only after selection, to
compute `use_this_index`; a mismatch set it to `false` and the whole
branch returned `None` (brute force). The iterator never advanced to a
second index on the same column whose metric did match. Results stayed
correct — the query just ran far slower than it needed to.

## Fix

Select over **all** indices keyed on the column with a known vector
type:

- Query specifies no metric → the first candidate answers, exactly as
before.
- Query specifies a metric → the first candidate whose metric equals it
answers.
- No candidate matches → the existing warn-and-brute-force path.

The metric is resolved through the existing fast path
(`metric_type_from_index_metadata`), so a modern index needs no I/O to
select; only legacy candidates without details are opened, and the loop
breaks on the first match. The segment-loading and coverage checks after
selection are unchanged.

## Tests

Added `test_knn_selects_index_matching_requested_metric`: build two
IVF_PQ indices on the same `vec` column (`idx_l2` with L2, `idx_cos`
with cosine), then assert via `explain_plan` that

- a **cosine** query hits `ANNSubIndex` with the cosine metric (index
used) rather than falling back to `KNNVectorDistance` (brute force), and
- an **L2** query still selects `idx_l2`.

Without the fix the cosine query brute-forces and the first assertion
fails. The existing `test_knn_metric_mismatch_falls_back_to_flat_search`
(single index, mismatched metric → brute force) and
`test_knn_no_metric_uses_index_metric` (no metric → index's own metric)
still pass, since both behaviours are preserved.

Also refactored the `TestVectorDataset` helper so
`make_vector_index_with_metric` delegates to a new
`make_named_vector_index_with_metric(name, metric)`, letting a test
place several differently-named indices on one column.

## Scope

Purely a selection/perf fix — no change to what any query returns. Only
the scanner's index-selection branch and a test helper are touched.

Signed-off-by: hello-peter-tang <132056702+hello-peter-tang@users.noreply.github.com>
…mat#9893)

## Summary
- Declare `json` and `from_json` on `CompactionPlan`, `CompactionTask`,
and `RewriteResult` so pyright no longer reports
`reportAttributeAccessIssue` for these existing native methods.
- Correct the stub property to `RewriteResult.original_fragments` to
match the native getter.
- Add a compaction typing regression to the pyright target and align the
Rust `RewriteResult.from_json` docstring with the stub. JSON stability
warnings are preserved.
- Resolve `CompactionOptions` through a relative import so pyright can
check the stub before the native extension is installed.

## Test plan
- [x] Full configured pyright target passes with pyright 1.1.414 in an
isolated environment without pylance: 0 errors, 14 warnings.
- [x] Incorrect `target_rows_per_fragment` argument types are still
rejected.
- [x] `uv run --no-sync make lint-python`: Ruff passes; pyright reports
0 errors and 15 dependency-stub/module-source warnings.
- [x] `uv run --no-sync pytest python/tests/test_optimize_typing.py
python/tests/test_optimize.py -k 'compaction_json_types or
dataset_distributed_optimize' --durations=30`: 8 passed. The new test
case takes 0.05 seconds.
- [x] Removing the six JSON method declarations and reverting the
property name produces 7 pyright attribute errors; the stub was restored
after this regression guard check.
- [x] The runtime `RewriteResult.from_json` docstring matches the stub.
- [x] `cargo fmt --all` passes for the workspace and Python crate.
- [ ] Rust clippy was not run locally because the
`no-local-cargo-builds` skill prohibits local Cargo lint builds.

---------

Co-authored-by: zhanghaobo@kanzhun.com <zhanghaobo@kanzhun.com>
…lance-format#9827)

Deleting a nonempty Lance branch on a TOS bucket with hierarchical
namespace can fail with HTTP 409 `CannotDelete`: the OpenDAL bridge
exposes directory entries as zero-size objects, and the generic deletion
stream may remove a directory before its children.

The generic `ObjectStore::remove_dir_all` now defers zero-size entries
and deletes them longest-path-first after nonempty files. Zero-byte
files are deferred too, which is safe because child paths are longer
than their parent directory. Nonempty files are forwarded through one
continuous `delete_stream`, retaining each provider's batching and
concurrency. All file deletes complete before deferred entries are
deleted. Cleanup uses a single wrapped listing, so tracing, metrics,
filtering, and path rewriting remain in effect; no native OpenDAL
listing or provider directory-operation refactor is needed.

Regression tests cover hierarchical and flat TOS listings, pagination
with each page read once, mixed nonempty and zero-byte files,
nested/empty/missing directories, encoded paths, sibling-prefix
isolation, deletion failures, rewritten paths, wrapped stores, and
dynamic credentials. A synthetic S3-style backend test verifies that
20,000 nonempty files retain 20 concurrent batches and that deferred
entries are deleted only after all file deletes finish.

Validation:
- `cargo test -p lance-io --features tos object_store::`: 277 passed.
- `cargo fmt --all` and `git diff --check`.
- `cargo clippy -p lance-io --features tos --tests --benches -- -D
warnings`.
- `cargo clippy --all --tests --benches -- -D warnings` (Rust 1.98.1).
- Reproduced the original failure on upstream
`4323cdee1374bfe28971a83f72dddf43f9cbc863` (14.0.0-beta.8). Revalidated
the revised fix on a disposable real TOS HNS dataset with two main
fragments / four rows and a branch appended to six rows: branch deletion
succeeds, its files and directories are removed, and the main dataset
still reads all four rows. The disposable dataset was fully cleaned up.

This fixes deletion ordering; it does not add rollback of an already
removed branch reference if cleanup fails.

CI status:
- [Lance Gatekeeper recommends approve without
changes](lance-format#9827 (review));
the empty-commit revision preserves that review.
- The earlier transitive `unsafe-libyaml` advisory blocker was addressed
upstream in [lance-format#9896](lance-format#9896).
The new CI run on `b31ed5953` passes the dependency audit and
license-header check. This PR adds no advisory exception.
- See the PR checks for the latest build and test results.

---------

Co-authored-by: wangzheyan <wangzheyan@bytedance.com>
… in accelerated IVF_PQ (lance-format#9755)

Fixes lance-format#9754

`create_index(..., index_type="IVF_PQ", metric="cosine",
accelerator=...)` built indices with much lower recall than the CPU
build (for example 0.387 vs 1.000 on 500k x 2048-d embeddings with
CUDA). The torch path trains the PQ codebook and computes the PQ codes
in Python, and the Rust builder uses them unchanged. For cosine the two
sides disagreed:

- Rust normalizes the vectors, takes residuals of the normalized
vectors, and treats the PQ codebook as an L2 codebook over them.
- The torch path trained the PQ codebook with cosine k-means, which
normalizes codewords and residual sub-vectors to unit length and so
drops the residual norms. It also took residuals of the unnormalized
vectors.

## Changes

All in `python/python/lance/vector.py`, cosine only:

- `train_pq_codebook_on_accelerator`: train the per-sub-vector k-means
with `l2`. PQ encoding in `one_pass_assign_ivf_pq_on_accelerator` goes
through the same `KMeans` objects, so it uses L2 as well.
- `compute_partitions` (residual sample for PQ training) and
`one_pass_assign_ivf_pq_on_accelerator` (final encoding): normalize the
vectors before subtracting the IVF centroid, as the Rust build does.

`l2` and `dot` are unchanged.

Not changed here: cosine IVF centroids from the torch k-means are still
normalized to unit length (lance-format#1759), while the CPU build keeps the mean of
the normalized vectors. Encoding and search both use the stored
centroids, so the index is consistent, but the residuals are larger.
This explains the remaining PQ-only gap in the table below: on the
pre-normalized 50k data, PQ-only recall@100 is 0.765 with this PR and
0.832 when the torch k-means also keeps unnormalized cosine centroids
(CPU build 0.827). With refine_factor=10 every variant is at 1.000. The
centroid change is in `lance.torch.kmeans.KMeans`, which is public and
also used by `IndicesBuilder`, so I left it for a follow-up.

## Tests

Added `test_torch_cosine_ivf_pq_recall` to
`python/python/tests/test_vector_index.py`, next to the other torch-path
index tests. It builds a cosine IVF_PQ index on 4096 x 32-d vectors with
varying norms, once without accelerator and once with
`accelerator=torch.device("cpu")`, and checks that the recall@10 of the
accelerated build is within 0.1 of the CPU build. It needs only torch
(`pytest.importorskip("torch")`), no GPU, and runs in about 3 s.

- Without the fix: CPU 0.840, torch 0.179 (fails).
- With the fix, 15 runs: torch recall 0.821 to 0.847, CPU recall 0.834
to 0.864, largest gap 0.035.

All Python numbers here come from the released pylance 11.0.0 wheel with
this `vector.py` change applied (the changed functions are identical on
`main`). I have not built `main` from source. On that setup the
torch-related tests of the 11.0.0 test suite (`test_vector_index.py`,
`test_indices.py`, `test_f16.py`, `torch_tests/`, filtered with `-k
"torch or accelerator or f16 or centroid or cuda or gpu"`) give 49
passed, 26 skipped, 1 failed both with and without the change. The
failure is
`torch_tests/test_data.py::test_bf16_to_tensor_clones_when_nulls_present`,
which does not use `vector.py`.

## Recall before / after

recall@100 against exact cosine top-100, 50k x 512-d synthetic data, 64
partitions, 64 sub-vectors, `accelerator="mps"`:

| Variant | Before: refine 10, nprobes 32 | Before: PQ only, all
partitions | After: refine 10, nprobes 32 | After: PQ only, all
partitions |
|---|---|---|---|---|
| CPU build, cosine | 1.000 | 0.835 | 1.000 | 0.827 |
| MPS, cosine, raw vectors | 1.000 | 0.502 | 1.000 | 0.768 |
| MPS, cosine, pre-normalized vectors | 0.690 | 0.110 | 1.000 | 0.765 |
| MPS, l2, pre-normalized vectors (reference) | 1.000 | 0.834 | 1.000 |
0.833 |

recall@10, 4096 x 32-d (script in the issue),
`accelerator=torch.device("cpu")`:

| Data | Before: CPU build | Before: torch build | After: CPU build |
After: torch build |
|---|---|---|---|---|
| raw vectors | 0.839 | 0.188 | 0.857 | 0.827 |
| pre-normalized vectors | 0.846 | 0.269 | 0.844 | 0.838 |

The CPU build is not seeded, so its numbers vary by about 0.02 between
runs.

Co-authored-by: Chastrlove <17813559+Chastrlove@users.noreply.github.com>
…ce-format#9701)

Description

The type stubs for the MemWAL classes in `lance/lance/__init__.pyi` had
drifted from the PyO3 signatures, so pyright both rejected valid calls
and accepted invalid ones.

| Method | Stub before | Binding |
|---|---|---|
| `_LsmScanner.limit` | `n` required | `n` defaults to `None`, so
`limit(offset=10)` is valid |
| `_LsmVectorSearchPlanner.plan_search` | no `refine_base_table` /
`overfetch_factor` | both supported |
| `_LsmVectorSearchPlanner.__init__` | no `filter` | `filter` supported
|
| `_ShardWriter.lsm_scanner` | accepts `None` | takes a list; `None`
raises `TypeError` at runtime |

`lance/mem_wal.py` is also added to pyright's `include` list, so future
drift in the first two kinds is caught in CI. The last two are not
visible through that file: it passes `filter` via `**kwargs` and never
passes `None` to `lsm_scanner`.

No runtime behavior changes.
…e-format#9693)

`LsmScanner::try_into_stream` returned JSON columns as raw JSONB
(`large_binary`), so Python `to_table` / `to_batches` disagreed with
`Dataset.to_table` and Java `scanBatches()`, which return Arrow JSON.

**Reading the same rows through different APIs gave different results.**

So convert in `try_into_stream`, as `Scanner::try_into_stream` does.
`count_rows` skips the conversion.
…ance-format#9641)

`execute_plan` and `analyze_plan` copy
`LanceExecutionOptions::batch_size` straight into the session config
with `options_mut()`, which bypasses the `batch_size > 0` assertion in
`SessionConfig::with_batch_size`. A batch size of 0 then reaches
DataFusion's operators: executing a `SortExec` with it panics with
`attempt to divide by zero` in `datafusion-physical-plan`
(`sorts/stream.rs:379`, DataFusion 54.1).

Return an `InvalidInput` error instead, before any plan runs. Dataset
scans already validate their batch size (`validate_batch_size`), so only
direct callers of `execute_plan` / `analyze_plan` /
`analyze_plan_with_context` are affected. `get_task_context` is private,
so no public signature changes.

This is also what DataFusion 55 requires: its `execution.batch_size`
becomes a `ConfigNonZeroUsize`, so the upgrade in lance-format#8997 only has to swap
the check for `ConfigNonZeroUsize::try_new`.

Added `test_zero_batch_size_is_rejected`, which checks both entry points
for the error variant and message. Ran `cargo fmt --all`, `cargo clippy
-p lance-datafusion --tests -- -D warnings`, and `cargo test -p
lance-datafusion --lib -- batch_size`.

Part of lance-format#9647.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ce-format#9639)

## Summary

`Scanner::nearest` used the entire child array when flattening a
`ListArray`
query. For sliced lists, this included values outside the visible slice,
causing
incorrect query dimensions or extra multivector query inputs.

Slice the child array using the first and last visible offsets,
preserving
zero-copy behavior and the existing query-count semantics.

## Validation

Regression tests compare sliced and equivalent list queries for flat and
indexed
search across multiple fragments, including prefix, middle, suffix, and
unsliced
cases. They also check multivector query keys and fixed-size-list query
handling.

- `cargo test -p lance --lib test_sliced_list_` — 2 passed.
- `cargo fmt --all` — passed; no changes.
- `cargo clippy --all --tests --benches -- -D warnings` — passed.
…ance-format#9637)

## Summary

When `n > max_takes` and no filter is given, `maybe_sample` takes `n //
max_takes` rows from each chunk. If `n` is not a multiple of `max_takes`
it returns fewer rows than requested (on a 10,000-row dataset with the
default `max_takes=2048`, asking for 3000 or 4095 rows returns 2500). If
`n` is a multiple, it stops after `max_takes` chunks and never samples
the tail of the dataset. This is the path used by
`lance.torch.data.LanceDataset(samples=N)`.

The fix samples each chunk in proportion to its size, so the counts add
up to exactly `n` and cover the whole dataset, and rounds the chunk size
up so the number of takes stays within `max_takes`.

## Test

Added `test_sample_dataset_returns_n_rows` to
`python/python/tests/test_sampler.py` (n = 100, 3000, 4095). It fails on
main (99, 2992, 4080 rows) and passes with the fix. `ruff format
--check` and `ruff check` pass on the changed files; pyright was not run
locally.
## Summary

`ShardedFragmentSampler` and the unfiltered path of
`ShardedBatchSampler` seed their shuffle with `seed` alone, so with
`randomize=True` every epoch yields the same order and `set_epoch()` has
no effect. `ShardedFixedBatchSampler` and the filtered
`ShardedBatchSampler` path already seed with `seed + epoch`; this makes
the other two do the same. Epoch 0 keeps its current order, and since
every rank uses the same `seed + epoch`, shards stay disjoint.

## Test

Added `test_set_epoch_reshuffles` to
`python/python/tests/test_sampler.py`, parametrized over both samplers.
It fails on main (identical order for epochs 1 and 2) and passes with
the fix. `ruff format --check` and `ruff check` pass on the changed
files; pyright was not run locally.
…rmat#9556)

`compact_files_with_planner` runs the rewrite tasks through
`buffer_unordered(num_threads)`. With `num_threads` set to `Some(0)`
that stream never pulls a task and returns `Pending` with nothing left
to wake it, so compaction neither finishes nor fails; in Java it also
keeps the dataset's write lock, so `close()` blocks too. The value
arrives unchanged from Python and Java, and zero often means "pick
automatically" elsewhere (rayon treats it that way).

Reject it once the plan is built, before any file is rewritten, which
also covers plans from a custom `CompactionPlanner`; `plan_compaction`
does not use the value and is unchanged. It is rejected rather than
replaced by the default, like the zero-parallelism guard lance-format#7632 added to
filtered reads. With nothing to compact, `Some(0)` now returns an error
instead of `Ok`.

From Java the error surfaces as `IllegalArgumentException` from
`compact()`.

## Testing

`cargo test --release -p lance test_compact_files_rejects_zero_threads`,
which times out on the old code, plus fmt and clippy with `-D warnings`.
…ance-format#9552)

`DatasetDeltaBuilder::build` rejects `compared_against_version` combined
with a begin/end range, but both bindings picked one mode before the
core saw them. Python's `LanceDataset.delta()` forwarded the range only
when `compared_against` was `None`, and the JNI `nativeBuild` forwarded
it only when `comparedAgainst` was unset. So `ds.delta(1,
begin_version=3, end_version=7)`, or the same selection on the Java
`DatasetDeltaBuilder`, returned the delta between version 1 and the
dataset's current version with no error, although the Python docstring
promises a `ValueError`.

Both bindings now forward every argument and let `build` reject the
combination.

## Behaviour change

Combining the two modes raises `ValueError` in Python and
`IllegalArgumentException` in Java instead of returning a delta. In
Java, setting only one end of the range still raises
`IllegalArgumentException`, now with the core's "Must specify both"
message instead of "Must specify either". `Dataset.delta(long)` and
`Dataset.delta(long, long)` set a single mode and are unaffected.

## Tests

`test_delta_validation_errors` and `DeltaTest` gain the conflicting
case, which returned a delta before the fix; `DeltaTest` also pins the
half-set message. Ran `pytest python/tests/test_delta.py` and `./mvnw
test -Dtest=DeltaTest`, plus `make lint-python`, spotless, and clippy
with `-D warnings` on `lance-jni`.
pjdurden and others added 28 commits October 27, 2026 18:18
…-format#8479)

Adds V2.0-V2.3 regression coverage for metadata-only all-null column addition followed by compaction, including ForceBinaryCopy rejection, safe re-encode fallback, row/order/null preservation, and persisted physical mapping.

Also corrects the Python BITMAP/LABEL_LIST row-address compatibility boundary so 15.0.0b2 is treated as a supporting reader while earlier versions retain full-scan fallback coverage.
…e-format#10350)

Streaming coreset training returned Float32 centroids for Float16 vector
columns. A complete IVF_RQ build then failed when residual computation
received those centroids alongside Float16 vectors.

Convert the final centroids back to Float16 after optional raw-vector
refinement, keeping training in Float32 and aligning the model with the
column for index construction and queries.

The regression builds one-bit cosine IVF_RQ indexes with 257 partitions
over two fragments, reopens the dataset, verifies the persisted centroid
type and index query plan, and asserts recall against brute-force
search. It probes all partitions and reranks the full 600-row fixture so
random one-bit quantization cannot discard true neighbors before
refinement. Cases cover both streaming samplers, omitted refinement,
Float32 streaming, and Float16 in-memory training. The fixed-sampler
Float16 cases reproduced the exact reported type mismatch before the
fix; both controls passed.

Validation:

- Five build-and-query regression cases pass in 20 consecutive runs (100
cases).
- `cargo test --profile ci -p lance --lib index::vector::ivf::tests::`:
123 passed.
- `cargo clippy --all --tests --benches --profile ci -- -D warnings`
passes.
- `cargo fmt --all -- --check` and `git diff --check` pass.

Fixes lance-format#10349

<!-- lance-gatekeeper-fix:v1 agent=af546cbd4568cbfd3cd4deb11866b7c4
generation=1 -->

---------

Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
…at#8400)

Preserve compaction row order across parallel, partial, and gapped rewrites while keeping manifests sorted by fragment ID.

Compaction results commit as one ordered replacement suffix. Metadata-only trailing relabels reserve fresh IDs, copy deletion files into the compacting dataset (including shallow clones), remap physical-address indexes, and retain the released table format. Row-adding concurrent writes still require replanning, while stale tasks can rebase across prior metadata-only relabels.

The strict `max_source_fragments` bound remains a documented, recoverable limitation tracked by lance-format#8986.

Fixes lance-format#3465
Bound indexed merge-insert execution with disk-backed source replay, capped spillable sort inputs, and a spillable sort-merge join.

Preserve provider snapshots, `FirstSeen` encounter order, partial-index fallback behavior, and stable row IDs across write batches and output fragments.

The accepted limits remain: low-cardinality index candidate maps do not spill, pipeline RSS is not a hard scan-wide cap, staging adds temporary I/O, and throughput is unbenchmarked.

Fixes lance-format#8129
…at#10268)

Bounds legacy take concatenation before Arrow offset capacity is exceeded, preserves duplicate rows, ordering, and carried-column alignment, and adds regression coverage for legacy row groups and indexed updates.
Fragment-scoped vector search now accepts the default postfilter path, combines selected ANN coverage with flat fallback, and constrains candidates before top-k. The change includes regression coverage for postfilters, fast search, deletions, overlays, stable row IDs, and fragment-reuse mappings.
Automatically batch reads of overflowed dictionaries under top-level primitive and struct decoders while preserving the recovery and exact page-merging semantics now on main.

The change adds capacity-aware batch limits for direct primitive and struct dictionary decoders and keeps stable historical regression coverage for Lance 2.0 and 2.1 plus pylance 10.0.0 and 13.0.0. Dictionaries nested under lists remain outside the automatic batching scope and may require an explicit smaller read batch. Decoder recovery and page concatenation retain `NotSupported` capacity errors.
## Why

ZoneMap V0 uses `max = +NaN` to mark any zone containing NaNs, including
negative NaNs. This hides the ordinary maximum and does not tell the
reader which NaN signs are present.

lance-format#10102 added ordinary bounds and separate negative-NaN counts to write
seeds. This PR brings the same information into **ZoneMap index V1**,
implementing the format proposed in lance-format#9986. It does not change the seed
format again.

## What V1 stores

For example, a zone containing `[1.0, 5.0, -NaN, +NaN]` is represented
as:

| Field | V0 | V1 |
| --- | --- | --- |
| `min` | `1.0` | `1.0` |
| `max` | `+NaN` marker | `5.0` |
| `nan_count` | `2` | `2` |
| `negative_nan_count` | Not stored | `1` |

V1 keeps these meanings separate:

- `min` / `max`: bounds on non-null, non-NaN values, following **Arrow
Rust ordering**. Infinities are included. Bounds are null when there are
no ordinary values; nested types retain their existing null-only
behavior.
- `null_count`: number of nulls.
- `nan_count`: number of all NaNs.
- `negative_nan_count`: number of those NaNs with the sign bit set.
Positive NaNs are `nan_count - negative_nan_count`.

The reader uses the bounds and NaN counts together for conservative
pruning. NaNs no longer overwrite an ordinary bound in a V1 file.

## What happens to existing indexes?

**New indexes use V1; existing V0 segments remain readable and are not
eagerly rewritten.** Both versions can serve the same logical index.
Each segment is decoded according to its own `index_version`.

| Operation | Behavior |
| --- | --- |
| Create a new segment | Write V1 |
| Dataset maintenance selects a V0 segment for replacement | Rebuild its
covered data as V1 |
| Low-level `ScalarIndex::update` | Preserve the input version,
following the existing convention |
| Direct merge of V0 + V0 | Write V0 |
| Direct merge of V1 + V1 | Write V1 |
| Direct merge of V0 + V1 | Reject before writing; rebuilding is
required |

A valid V0 seed is eligible for V1 reuse only when its stored
`nan_count` is explicitly zero; its negative count is then provably
zero. Missing counts are not zero. A V0 seed containing NaNs cannot
supply V1's ordinary maximum and exact sign split, so the update falls
back to scanning rather than inventing statistics. Every output V1
record must satisfy the V1 contract; conservative V0 query compatibility
does not relax it. This PR retains coarse-grained fallback;
finer-grained seed reuse is separate work.

Optional V0 seeds with invalid counts or incomplete fragment coverage
also fall back to scanning before reaching the strict V1 writer.

## Safety and compatibility

- V1 read/write validation rejects impossible counts and NaN bounds. A
retained, shortened FRI-remapped zone is also rejected by direct merge
when its counts can no longer be established exactly; callers must
rebuild it instead. Destination fragments excluded from the merge are
filtered before fallible remap-run construction and validation.
- The existing `index_version` filter lets readers without V1 support
skip V1 segments and fall back to scans. New readers continue to read V0
segments under their original contract.
- No new table feature flag is added. Zone sizing, configurable
null-bitmap collection, and packed in-memory storage are outside this
PR.

## Tests

Coverage includes V0/V1 round trips, invalid statistics, unknown
versions, seed reuse/fallback, merge-version rules, remap-count safety,
a released V8 fixture rebuilt as V1, and fresh V1 segments committed
alongside untouched V0 coverage. Dataset tests assert the index is used
and results equal a scan.

Validation at `f497c6c97`:

- GitHub checks completed with no failures, including Linux x86/ARM,
macOS, Windows and compatibility tests. The external-plugin test fixture
now advertises the ZoneMap version it delegates to, fixing the earlier
four-platform CI failure.
- Author-run fmt and clippy passed; `lance-index` 2,775 passed / 3
ignored. Targeted ZoneMap tests: 68 lance-index and 117 Lance tests
passed; external-plugin integration tests: 7 passed.
- Author-run Lance library: 6,402 passed / 1 failed / 4 ignored. The
failure was the FTS background tiered-merge timing test; all five
isolated reruns passed. That local full invocation was **not entirely
green**. Other integration suites and doc tests passed.
- Author rebuilt Python bindings on the same head: targeted tests 25
passed; compatibility 7 passed / 1 skipped (the suite excludes ZoneMap
0.x downgrade); Python lint passed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Pin the pip-installed Python test extra to pytest 8.4.2 and pytest-xdist 3.8.0, matching the existing uv test group.

The macOS CI jobs resolved pytest 9.2.0 before this change. With that runner, the seven lindera fixture tests were distributed across workers and raced on the shared extracted dictionary. The corrected head installs pytest 8.4.2, serializes all seven tests with `@lindera`, and passes the complete Python workflow (1,895 passed, 530 skipped on macOS).
Record the vector-per-partition target implied by an explicit partition count or centroids for full-table IVF builds. The target is derived from the build's actual shuffled vector count and effective trained partition count, then carried through index details, append, optimize, and retrain. Explicit positive targets still win; fragment subsets, multivector builds, and zero-vector builds retain the default; explicit zero targets are rejected with or without a partition count or centroids.

The new public `VectorIndexBuildSummary::num_vectors_shuffled` field requires the breaking marker.

Validation: current-head Rust, Python, Java, module-size, license, typo, format gate, and repository checks passed after merging the latest main and the compatible pytest dependency pin. The Python macOS suite passed 1,895 tests with 530 skipped. The targeted 13 Rust tests and three Python retrain cases passed at 484798b; Gatekeeper approved with the documented retrain risk and all threads are resolved.
…e-format#10387)

## Summary

The quickstart only showed PyLance, but most users are better served by
a fuller engine. This adds LanceDB and Apache Spark quickstarts next to
PyLance, modeled on how the Iceberg site offers one quickstart per
engine. LanceDB becomes the landing page, so the site's "Get started"
buttons now go there.

```text
Getting Started
├── LanceDB          quickstart/            (new; was the PyLance intro)
├── Apache Spark     quickstart/spark/      (new; SQL + PySpark tabs)
└── PyLance
    ├── Getting Started   quickstart/pylance/   (moved from quickstart/)
    ├── Versioning        (URLs unchanged)
    ├── Vector Search
    └── Full-Text Search
```

Both new pages walk through the same product catalog: create a table,
filter it, run vector and full-text search, and modify it. LanceDB adds
time travel. Spark ends by opening the table it wrote from LanceDB, to
show that the engines share one format.

The page code is not inline Markdown. It is embedded with
`pymdownx.snippets` from runnable files in `docs/snippets/quickstart/`,
and each step's output is shown below its code. The output comes from
files in `outputs/` that a new `docs-snippets` workflow checks against a
fresh run:

- Python snippets run section by section like notebook cells. Each
section's printed output and the value of its final expression are
recorded.
- The Spark SQL file runs through the exact `spark-sql` command the page
shows. Marker queries split its output by section.
- If the output drifts, CI fails with a diff. `pytest --update-outputs`
re-records it.
- Vectors use multiples of 0.125 so distances are exact in float32, and
pandas output is rounded to 4 decimals, so recorded output doesn't vary
by platform.

The workflow also checks that the `pip install` commands on the pages
accept the versions under test. `check_paths: true` makes the docs build
fail on a broken snippet reference instead of rendering an empty block.

Pins and caching:

- Python deps are pinned in `docs/snippets/pyproject.toml` and
`uv.lock`, and cached by `setup-uv`. I added a `!snippets/uv.lock`
exception to `docs/.gitignore`, which otherwise ignored the lockfile.
- The Spark connector bundle (`lance-spark-bundle-4.2_2.13:0.8.0`) pulls
about 577 MB of jars. They are cached under a key hashed from the files
that name the coordinate, with a prefix fallback so a version bump
downloads only what changed.
- Dependabot bumps the Python pins monthly, in one grouped PR with a
2-day cooldown. The connector coordinate is bumped by hand: a PySpark
minor bump fails CI until it is updated, and that failure serves as the
reminder.

## Evidence

Locally (JDK 17, Python 3.12), all five snippet tests pass in about 30s
with warm jars. The full-site build, run with lance-spark's docs
assembled in, shows no new warnings. The links from `spark.md` to
`../integrations/spark/...` only resolve in that assembled build, like
the other integration links.

## Merge Risk

**Door:** two-way. Docs, CI, and Dependabot config only. The one URL
change: `quickstart/` now shows LanceDB instead of the PyLance intro,
which moved to `quickstart/pylance/`.
**Blast Radius:** docs site, CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
UpdateBuilder::set rejected any column containing blob v2 values, so an
UPDATE could carry blob columns through unchanged but never assign one.
A column holding blob v2 values at the top level or inside structs,
lists or large lists now takes an expression that yields bytes in their
place: the bytes are wrapped in the logical blob layout before the write
path, which stores them inline, packed or dedicated as usual.

The rewrite feeds the assigned blob values in as logical placeholders,
found from the dataset schema, so it no longer reads old blobs that the
assignment overwrites, and it keeps the other fields of a struct. An
update that reads a blob column it also assigns is rejected. Blob v2
values inside maps and fixed-size lists stay unsupported: Lance cannot
write the former, and its file reader cannot read a fixed-size list of
structs back.
…ow (lance-format#10339)

Constant pages (2.2+) were decoded one row at a time. `skip_to_row`
stepped through every row before the requested one, and `drain` called
`take_row` and recounted visibility once per row. Each take builds a new
decoder whose cursor starts at the head of the page, so reading a single
row near the end of a page walked the whole page.

Without repetition levels every row is exactly one level, so jump the
cursor straight to the target row, as the complex all-null decoder
already does. Each drained range is then one level slice. `drain` only
counts visible items when there are no definition levels; when there
are, `decode` already counts them from the sliced levels. Pages with
lists still step row by row to find the range ends. The decoded values
are now wrapped with `DataBlock::from_decoded_array`, which skips
encoder statistics and reuses the fixed-width buffer instead of copying
it.

Output is unchanged. The existing constant-layout tests now also read
ranges and takes that start mid-page.

**Measured on a 2.2 file with a single-value int64 column, pinned to 16
cores, best of 5**, two interleaved rounds:

| Case | Before | After |
| --- | ---: | ---: |
| scan 4M rows | 13.4-14.3 ms | 3.5-3.6 ms |
| scan 1M rows | 2.4-2.6 GB/s | 16.4-17.8 GB/s |
| take 100 rows out of 1M | 2.05 ms | 0.01 ms |
…t#10393)

lance-format#9977 fixed DynamoDB key-type validation and paginated manifest deletion
for
lance-format#9974, which is now closed. This test-only follow-up preserves the three
cases
that were unique to duplicate PR lance-format#9980:

- A 26-row query page is split into exact 25-key and 1-key
  `BatchWriteItem` requests, with the full request sequence and key sets
  asserted.
- Continuously returned `UnprocessedItems` are re-sent until the
production
bound is exhausted. The test asserts the initial send plus eight
re-sends,
the explicit `Error::IO` and diagnostic text, and that later requests
contain
  only the remaining key.
- Query responses with missing `Items` and with `Items: []` both
complete
  successfully without issuing `BatchWriteItem`.

The tests use the existing scripted endpoint and real AWS SDK path; no
production code or retry configuration changes are included. Dataset
deletion
automatically cleaning DynamoDB manifest rows remains lance-format#3494 and is
outside this
change.

The retry-exhaustion test drives all nine batch requests through paused
Tokio
time and the real SDK to a local scripted endpoint. It completed in
0.13s
without hitting an SDK network timeout.
Closes lance-format#9647.

DataFusion 55 and Arrow 59 require a coordinated upgrade across the Rust
workspace, Python bindings, and Java JNI crate. This PR carries the
original implementation forward, resolves its conflicts with the current
`main`, and adapts the remaining execution-plan, spill, and binding APIs
to DataFusion 55.

This replaces lance-format#9979 because that branch is owned by an organization
where the maintainer cannot push. The original commit authors are
preserved in the branch history; the trailers at the end of this
description also retain their credit if the PR is squash-merged.

### Behavior changes

- Migrate execution-plan expression reporting, statistics, planning
contexts, optimizer composition, and `AnalyzeExec` construction to
DataFusion 55.
- Preserve full-text-search ordering during the `EnforceDistribution`
removal, align indexed and hash-join signed-zero matching, and keep
Arrow 59 map copying behavior compatible with existing data.
- Use DataFusion 55's public `SpillFile` API and refresh the Rust,
Python, Java, and Python dependency lockfiles.
- Package the local Arrow helper crates in CI while their `59.0.0`
versions are not yet on crates.io.

### Verification

- `cargo fmt --all -- --check`
- `cargo clippy --all --tests --benches -- -D warnings`
- `cargo check --workspace --tests --benches`, including
`--all-features`
- `cargo package --workspace --no-verify --all-features`
- Targeted Rust tests for DataFusion, spill handling, merge-insert
behavior, Arrow ordering, count pushdown, top-k late materialization,
full-text search, and execution plans
- Python `make install`, `uv run make lint`, and
`test_table_provider.py`
- Java JNI clippy, Spotless, and `./mvnw test`

Thanks to @timsaucer for the original implementation and careful
behavior analysis, and to @slachiewicz, @wjones127 for the review,
signed-zero discussion, and follow-up coverage. Their contributions and
review history made this upgrade possible.

Co-authored-by: Sylwester Lachiewicz <slachiewicz@apache.org>
Co-authored-by: Tim Saucer <timsaucer@gmail.com>

---------

Co-authored-by: Tim Saucer <timsaucer@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Sylwester Lachiewicz <slachiewicz@apache.org>
…concat (lance-format#9912)

Preserve add-column stream sub-batches through the updater so large variable-width outputs do not require a 32-bit-offset concat. Deleted-row alignment and legacy row-group behavior remain unchanged.
…9869)

Delete data, deletion, and owned managed Blob payloads written by merge_insert analyze plans without committing, while preserving existing fragments and reused Blob references.

Closes lance-format#9862.
…ance-format#10406)

Wait for asynchronous replay-spill cleanup before asserting quota release

This is a test-only synchronization fix for the intermittent DataFusion spill quota check. It preserves the assertion that disk reservations must eventually reach zero while allowing the background drain task to finish cleanup; library behavior is unchanged.
…mespace (lance-format#9867)

Implement DirectoryNamespace create/drop mode semantics and require existing parent namespaces before table creation or declaration.

The final integration preserves namespace-specific lookup, object-collision checks before marker writes, canonical table locations, and root-equivalent drop protection. Destructive overwrite and cascade modes remain explicitly unsupported.
…rmat#10413)

Nullable struct merges can panic on a sliced Boolean child because the
rebuilt null bitmap does not cover the child array offset. Align the
merged validity bitmap with the child offset before rebuilding the
array, preserving the existing buffer when the offsets already match.

Add 84 regression combinations over child/parent bit offsets 0, 1, 7, 8,
9 and 1025, with and without child nulls. The expected logical values
are independent of the bitmap representation. This completes the
sliced-child case left uncovered by lance-format#9503; related downstream issue:
[LanceDB lance-format#4260](lancedb/lancedb#4260).

Validation:
- Before the fix: 35 of the 84 new cases fail on main.
- `cargo test -p lance-arrow`: 203 unit tests and 6 doctests pass after
the fix.
- `cargo fmt --all` and `cargo clippy --all --tests --benches -- -D
warnings` pass.

The equivalent Arrow 58 backport for the planned 14.0.2 patch release is
lance-format#10412. It is being validated against freshly fetched LanceDB main,
without the downstream vendor override.
…rmat#9769)

Record metadata-only suffix relabels with all moved physical row addresses, including deleted rows. Integrate strict trailing-suffix planning with recorded and unrecorded compaction races, and cover the resulting Rust and Python paths. Keep the accepted lance-format#9947 follow-up outside this PR scope.
…mat#9721)

## Summary
- Follow-up to lance-format#9518 (merged). Addresses the non-blocking risk the
gatekeeper flagged there: `JsonIndexPlugin::load_index` passed
`target_plugin.version()` (the target plugin's current max format
version) instead of the version the target index was actually persisted
at, since `JsonIndexDetails` never recorded it.
- Adds an optional `target_index_version` field (field 4) to
`JsonIndexDetails`, populated from `CreatedIndex::index_version`
whenever the wrapper trains, remaps, or updates its target. `load_index`
now reads it, falling back to the target plugin's current version only
for details written before this field existed (matching the previous,
safe-for-now behavior for old data).
- Drops the now-stale note in the `target_data_type` docs saying these
details do not record the target's format version.
- This matters for the stacked [BTree format
change](lance-format#9481), which will use
the passed-in version to distinguish stored row IDs from row addresses.

## Test plan
- `cargo clippy -p lance-index -p lance --tests -- -D warnings` clean
- `cargo fmt --all -- --check` clean
- `cargo test -p lance-index --lib scalar::json::` — 27 passed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018BsnoP9ZPMX6vhJSygzdnN

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…10417)

Follow up on [the review comment in
lance-format#10413](lance-format#10413 (comment)).

A typed `NullBuffer` preserves its own bit offset. Arrow 59.3 validates
its backing byte length against `ceil((child.offset() + child.len()) /
8)`, so different offsets do not always require bitmap realignment.
Reuse a buffer that already meets that bound and retain the existing
padding fallback for an undersized buffer.

Add four cases covering offsets within one byte, different offsets, and
small slices with large child or parent offsets. They verify logical
null values, shared backing storage, full ArrayData validation and IPC
round trips. The four storage-sharing checks fail before this change and
pass afterward.

Validation:
- `cargo test -p lance-arrow`: 207 unit tests and 6 doctests passed,
including the existing 84 offset regressions.
- `cargo fmt --all` and `cargo clippy --all --tests --benches -- -D
warnings` passed.
- Buffer reuse is verified through pointer identity; throughput and RSS
were not benchmarked.

This change targets main's typed-null-buffer path. The validated 14.0.2
RC uses the separate raw-bitmap backport from lance-format#10412.
@sezruby

sezruby commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Merged upstream in lance-format#9638.

@sezruby sezruby closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.