Skip to content

fix(core): narrow LargeList children in schema intersection - #9503

Merged
Xuanwo merged 16 commits into
lance-format:mainfrom
LuciferYang:fix/intersection-large-list
Oct 23, 2026
Merged

Xuanwo merged 16 commits into
lance-format:mainfrom
LuciferYang:fix/intersection-large-list

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

A large_list<struct<...>> column cannot be read once a fragment is missing one of the struct's children, as happens after an append whose batch leaves a child out. The append succeeds, and every later read of that column fails with projection supplied fewer column indices than its fields require. The same steps on a list<struct<...>> column fill the missing child with nulls.

Field::do_intersection picks the fields it recurses into with a matches! that lists Struct, List and Map but not LargeList, so two LargeList fields fall through to the type-equality branch: the strict form errors, and the ignore_types form returns the whole unnarrowed field. The fragment reader uses the ignore_types form to project the dataset schema down to what a file holds, so the older file is asked for the child it never wrote. Two other matches! in the same file already pair (LargeList, LargeList) with List; this adds the missing pair. The batch merge that runs next already takes a LargeList of structs down the List arm since #9954.

That merge arm did not push a null parent row down into the list, which #7494 noted for List, and neither did the arm for a struct inside a struct. A merged struct row is valid if either side is, so a column sitting under the null side became visible with whatever values it held. Every column the merge takes from one side, or recurses into, now first gets that side's null parent rows pushed in, at any depth. When two lists with different item structs are merged entry by entry, an entry is valid if either side's is, and a side's items under an entry that only the other side has are nulled first. A side whose entries are all hidden by null parents is merged as the all-null placeholder, so the other side keeps its offsets. A non-nullable field that would have to show a hidden row, because only the other side has that row, makes the merge return an InvalidArgumentError naming the field instead of the hidden value. adjust_child_validity, which does the pushing down, also lost the bit offset of a sliced parent's validity and nulled the wrong rows; it now takes the union of the two null buffers.

The intersection recursion can also end with no children at all, because Arrow does not fix the name of a list's item child or a map's entries child: either the other field has no child of that name, or the recursive intersection failed and .ok()? dropped it. For List, LargeList and Map such a field cannot be turned back into a DataType, because Field::data_type() indexes children[0] for all three. For List and Map it used to be returned successfully and panic at the next type access; it is now an Error::arrow naming both sides, and LargeList, which main never recursed into, gets the same error. Struct is exempt: data_type() maps over the children there, so an empty struct is representable.

Several behaviour changes are not specific to LargeList. A zero-children List or Map intersection used to panic later and now fails at the intersection, and intersection_ignore_types on two LargeLists whose item children are named differently, which main returned unnarrowed, now fails the same way. A List of structs or a nested struct under a null parent row is now null after a merge, a sliced parent's nulls now land on the right rows, and a non-nullable field under a parent row only one side has is an error rather than an exposed value or a panic. A merged list's item field is nullable if either side's is, in the placeholder case as well. And under strict intersection, a LargeList whose nested subfield types conflict now has that subfield dropped by .ok()? rather than failing the whole field, which is how List already behaves.

Fixes #9502.

How was this patch tested

test_insert_subschema_nested_in_a_list appends a batch missing one struct child to a list-of-struct column and reads the column back, for List and LargeList across V2.0, V2.1 and V2.2. Reverting the (LargeList, LargeList) pair in do_intersection turns the LargeList cases red; the List cases pass either way.

test_intersection in schema.rs gains a LargeList of struct narrowed through intersection_ignore_types in both directions and through the strict intersection. Deleting the (LargeList, LargeList) pair makes that block fail.

test_list_field_intersection_without_shared_children is an rstest table over four cases where the recursion ends with no children: List and LargeList with mismatched item names, LargeList with matching item names but conflicting item types, and Map with mismatched entries names. Each asserts Error::Arrow and the guard's message. test_struct_field_intersection_without_shared_children pins the struct exemption from the other side.

In lance-arrow, these merge tests fail on main. test_merge_list_struct_hides_the_items_of_a_null_parent_row puts the null parent row on either side and test_merge_list_struct_follows_a_sliced_parent does it after slicing, both for List and LargeList. test_merge_hides_nested_structs_of_a_null_parent_row checks a struct nested in a list item, for both widths, and in a plain struct. test_merge_list_struct_identical_schema_hides_a_null_parent_row checks the same when both sides have the same item type. test_merge_list_struct_with_non_nullable_fields_under_a_null_parent, over List, expects an error where a non-nullable field would show a hidden row and a clean merge otherwise. test_merge_list_struct_hidden_side_keeps_the_other_offsets merges a list whose entries are all under null parents with one of different offsets and checks the other side's lists survive; it fails if the placeholder side is picked before the parents are applied. Two keep the reader's placeholder path working and pass on main: test_merge_list_struct_under_null_parents_with_a_placeholder merges a batch whose parent rows are all null with an all-null placeholder, and test_merge_with_a_placeholder_under_a_non_nullable_field merges a placeholder under a non-nullable struct, list and large list in both orders.

`Field::do_intersection` recursed into children for Struct, List, and
Map pairs but not LargeList, so intersecting two large_list fields fell
through to the type-equality check and returned the full field instead
of the shared subfields. After nested schema evolution under a
LargeList column, `Fragment::open_*_reader` intersects the requested
projection with each data file's schema; the un-narrowed result then
failed `project_by_schema(OnMissing::Error)` with a missing-field
error even though the read should succeed with the shared columns.
Intersecting two list-like fields whose item children all fail the
name match (or whose recursive item intersection errors) returned a
zero-children field. Field::data_type indexes children[0] for
List/LargeList/Map, so materializing such a field panics instead of
surfacing an error. Return a descriptive Error::arrow instead; at
nested levels the existing .ok()? swallow drops the field from its
parent, and at the top level the error fails the intersection.
Turn the rejection test into an rstest case table asserting the error
variant and message, report both sides in the guard's error instead of
only self, and pin the deliberate Struct exemption with its own test.
@github-actions github-actions Bot added the bug Something isn't working label Sep 23, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 23, 2026
…s merged

Narrowing the projection is half of reading a struct child an older file
does not have: the reader then merges that file's batch with the nulls for
the missing child, and that merge only recursed into List<Struct>, so a
LargeList<Struct> kept the older file's struct and the read failed with
"field b does not exist in the RecordBatch". Adds the dataset-level append
and read that shows both halves.
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Oct 16, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Oct 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Oct 16, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Oct 16, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 16, 2026
@wjones127 wjones127 added the critical-fix Bugs that cause crashes, security vulnerabilities, or incorrect data. label Oct 19, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 22, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 22, 2026
@LuciferYang

Copy link
Copy Markdown
Contributor Author

@wkalt could you take a look, since it touches the list-of-struct merge you restructured in #9954? A large_list<struct> column becomes unreadable after an append whose batch leaves a struct child out, because schema intersection skips LargeList, and the merge shows values sitting under a null parent row. I merged main and layered the parent hiding onto #9954's structure.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 22, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 23, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

The null-parent finding remains fixed after merging main: values under null ancestors stay hidden, and unrepresentable non-nullable merges return errors. The nested-list behavior from #9954, visible offsets, and item nullability are preserved. The LargeList append/read, recursive schema-evolution, and projected missing-child scan/take regressions pass on this head.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 23, 2026
@github-actions github-actions Bot added A-python Python bindings A-java Java bindings + JNI A-deps Dependency updates labels Oct 23, 2026
@Xuanwo
Xuanwo force-pushed the fix/intersection-large-list branch from d3ebee1 to 3f4d03a Compare October 23, 2026 09:21
@Xuanwo
Xuanwo merged commit 46e3ff5 into lance-format:main Oct 23, 2026
40 checks passed
@LuciferYang

Copy link
Copy Markdown
Contributor Author

Thank you @Xuanwo

BubbleCal added a commit that referenced this pull request Oct 28, 2026
Nullable struct merges can panic on a sliced Boolean child because the
rebuilt null bitmap does not cover the child array offset. Align the
merged validity bitmap with the child offset before rebuilding the
array, preserving the existing buffer when the offsets already match.

Add 84 regression combinations over child/parent bit offsets 0, 1, 7, 8,
9 and 1025, with and without child nulls. The expected logical values
are independent of the bitmap representation. This completes the
sliced-child case left uncovered by #9503; related downstream issue:
[LanceDB #4260](lancedb/lancedb#4260).

Validation:
- Before the fix: 35 of the 84 new cases fail on main.
- `cargo test -p lance-arrow`: 203 unit tests and 6 doctests pass after
the fix.
- `cargo fmt --all` and `cargo clippy --all --tests --benches -- -D
warnings` pass.

The equivalent Arrow 58 backport for the planned 14.0.2 patch release is
#10412. It is being validated against freshly fetched LanceDB main,
without the downstream vendor override.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates A-java Java bindings + JNI A-python Python bindings bug Something isn't working critical-fix Bugs that cause crashes, security vulnerabilities, or incorrect data. K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: schema intersection does not narrow LargeList children, and can return a list field that panics on data_type()

3 participants