Repository navigation
fix(core): narrow LargeList children in schema intersection - #9503
Conversation
`Field::do_intersection` recursed into children for Struct, List, and Map pairs but not LargeList, so intersecting two large_list fields fell through to the type-equality check and returned the full field instead of the shared subfields. After nested schema evolution under a LargeList column, `Fragment::open_*_reader` intersects the requested projection with each data file's schema; the un-narrowed result then failed `project_by_schema(OnMissing::Error)` with a missing-field error even though the read should succeed with the shared columns.
Intersecting two list-like fields whose item children all fail the name match (or whose recursive item intersection errors) returned a zero-children field. Field::data_type indexes children[0] for List/LargeList/Map, so materializing such a field panics instead of surfacing an error. Return a descriptive Error::arrow instead; at nested levels the existing .ok()? swallow drops the field from its parent, and at the top level the error fails the intersection.
Turn the rejection test into an rstest case table asserting the error variant and message, report both sides in the guard's error instead of only self, and pin the deliberate Struct exemption with its own test.
…s merged Narrowing the projection is half of reading a struct child an older file does not have: the reader then merges that file's batch with the nulls for the missing child, and that merge only recursed into List<Struct>, so a LargeList<Struct> kept the older file's struct and the read failed with "field b does not exist in the RecordBatch". Adds the dataset-level append and read that shows both halves.
…rge-list # Conflicts: # rust/lance-arrow/src/lib.rs
|
@wkalt could you take a look, since it touches the list-of-struct merge you restructured in #9954? A |
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The null-parent finding remains fixed after merging main: values under null ancestors stay hidden, and unrepresentable non-nullable merges return errors. The nested-list behavior from #9954, visible offsets, and item nullability are preserved. The LargeList append/read, recursive schema-evolution, and projected missing-child scan/take regressions pass on this head.
d3ebee1 to
3f4d03a
Compare
|
Thank you @Xuanwo |
Nullable struct merges can panic on a sliced Boolean child because the rebuilt null bitmap does not cover the child array offset. Align the merged validity bitmap with the child offset before rebuilding the array, preserving the existing buffer when the offsets already match. Add 84 regression combinations over child/parent bit offsets 0, 1, 7, 8, 9 and 1025, with and without child nulls. The expected logical values are independent of the bitmap representation. This completes the sliced-child case left uncovered by #9503; related downstream issue: [LanceDB #4260](lancedb/lancedb#4260). Validation: - Before the fix: 35 of the 84 new cases fail on main. - `cargo test -p lance-arrow`: 203 unit tests and 6 doctests pass after the fix. - `cargo fmt --all` and `cargo clippy --all --tests --benches -- -D warnings` pass. The equivalent Arrow 58 backport for the planned 14.0.2 patch release is #10412. It is being validated against freshly fetched LanceDB main, without the downstream vendor override.
A
large_list<struct<...>>column cannot be read once a fragment is missing one of the struct's children, as happens after an append whose batch leaves a child out. The append succeeds, and every later read of that column fails withprojection supplied fewer column indices than its fields require. The same steps on alist<struct<...>>column fill the missing child with nulls.Field::do_intersectionpicks the fields it recurses into with amatches!that listsStruct,ListandMapbut notLargeList, so twoLargeListfields fall through to the type-equality branch: the strict form errors, and theignore_typesform returns the whole unnarrowed field. The fragment reader uses theignore_typesform to project the dataset schema down to what a file holds, so the older file is asked for the child it never wrote. Two othermatches!in the same file already pair(LargeList, LargeList)withList; this adds the missing pair. The batch merge that runs next already takes aLargeListof structs down theListarm since #9954.That merge arm did not push a null parent row down into the list, which #7494 noted for
List, and neither did the arm for a struct inside a struct. A merged struct row is valid if either side is, so a column sitting under the null side became visible with whatever values it held. Every column the merge takes from one side, or recurses into, now first gets that side's null parent rows pushed in, at any depth. When two lists with different item structs are merged entry by entry, an entry is valid if either side's is, and a side's items under an entry that only the other side has are nulled first. A side whose entries are all hidden by null parents is merged as the all-null placeholder, so the other side keeps its offsets. A non-nullable field that would have to show a hidden row, because only the other side has that row, makes the merge return anInvalidArgumentErrornaming the field instead of the hidden value.adjust_child_validity, which does the pushing down, also lost the bit offset of a sliced parent's validity and nulled the wrong rows; it now takes the union of the two null buffers.The intersection recursion can also end with no children at all, because Arrow does not fix the name of a list's item child or a map's entries child: either the other field has no child of that name, or the recursive intersection failed and
.ok()?dropped it. ForList,LargeListandMapsuch a field cannot be turned back into aDataType, becauseField::data_type()indexeschildren[0]for all three. ForListandMapit used to be returned successfully and panic at the next type access; it is now anError::arrownaming both sides, andLargeList, which main never recursed into, gets the same error.Structis exempt:data_type()maps over the children there, so an empty struct is representable.Several behaviour changes are not specific to
LargeList. A zero-childrenListorMapintersection used to panic later and now fails at the intersection, andintersection_ignore_typeson twoLargeLists whose item children are named differently, which main returned unnarrowed, now fails the same way. AListof structs or a nested struct under a null parent row is now null after a merge, a sliced parent's nulls now land on the right rows, and a non-nullable field under a parent row only one side has is an error rather than an exposed value or a panic. A merged list's item field is nullable if either side's is, in the placeholder case as well. And under strictintersection, aLargeListwhose nested subfield types conflict now has that subfield dropped by.ok()?rather than failing the whole field, which is howListalready behaves.Fixes #9502.
How was this patch tested
test_insert_subschema_nested_in_a_listappends a batch missing one struct child to a list-of-struct column and reads the column back, forListandLargeListacross V2.0, V2.1 and V2.2. Reverting the(LargeList, LargeList)pair indo_intersectionturns theLargeListcases red; theListcases pass either way.test_intersectioninschema.rsgains aLargeListof struct narrowed throughintersection_ignore_typesin both directions and through the strictintersection. Deleting the(LargeList, LargeList)pair makes that block fail.test_list_field_intersection_without_shared_childrenis an rstest table over four cases where the recursion ends with no children:ListandLargeListwith mismatched item names,LargeListwith matching item names but conflicting item types, andMapwith mismatched entries names. Each assertsError::Arrowand the guard's message.test_struct_field_intersection_without_shared_childrenpins the struct exemption from the other side.In
lance-arrow, these merge tests fail on main.test_merge_list_struct_hides_the_items_of_a_null_parent_rowputs the null parent row on either side andtest_merge_list_struct_follows_a_sliced_parentdoes it after slicing, both forListandLargeList.test_merge_hides_nested_structs_of_a_null_parent_rowchecks a struct nested in a list item, for both widths, and in a plain struct.test_merge_list_struct_identical_schema_hides_a_null_parent_rowchecks the same when both sides have the same item type.test_merge_list_struct_with_non_nullable_fields_under_a_null_parent, overList, expects an error where a non-nullable field would show a hidden row and a clean merge otherwise.test_merge_list_struct_hidden_side_keeps_the_other_offsetsmerges a list whose entries are all under null parents with one of different offsets and checks the other side's lists survive; it fails if the placeholder side is picked before the parents are applied. Two keep the reader's placeholder path working and pass on main:test_merge_list_struct_under_null_parents_with_a_placeholdermerges a batch whose parent rows are all null with an all-null placeholder, andtest_merge_with_a_placeholder_under_a_non_nullable_fieldmerges a placeholder under a non-nullable struct, list and large list in both orders.