Skip to content

GH-50847: [Python] Add Type_RUN_END_ENCODED to _NESTED_TYPES set - #50848

Open
Nishuuzz wants to merge 1 commit into
apache:mainfrom
Nishuuzz:gh-50847-is-nested-ree
Open

GH-50847: [Python] Add Type_RUN_END_ENCODED to _NESTED_TYPES set#50848
Nishuuzz wants to merge 1 commit into
apache:mainfrom
Nishuuzz:gh-50847-is-nested-ree

Conversation

@Nishuuzz

@Nishuuzz Nishuuzz commented Aug 11, 2026

Copy link
Copy Markdown

Rationale for this change

pyarrow.types.is_nested() said a run-end encoded type wasn't nested, while the C++ arrow::is_nested() says it is:

>>> import pyarrow as pa
>>> t = pa.run_end_encoded(pa.int32(), pa.string())
>>> t.num_fields
2
>>> pa.types.is_nested(t)
False

The Python side reads from a hardcoded _NESTED_TYPES set in python/pyarrow/types.py rather than from the C++ trait, so it has to be kept in step by hand. Lining that set up against is_nested() in cpp/src/arrow/type_traits.h, run-end encoded was the only type the two still disagreed about — the list-view types and fixed-size list are already there.

It's also out of step with the type itself. Run-end encoded has two children, the run ends and the values, and every other type in pyarrow that has children answers True here.

This is the same thing that happened to fixed-size list in #40171, fixed by #40172, and the list-view types were added after that. This looks like the last one missed when run-end encoding went in.

What changes are included in this PR?

Adds Type_RUN_END_ENCODED to _NESTED_TYPES, which is the whole fix.

In the test I've added the run-end encoded case, and while I was there also a map and a dictionary. Map was nested already but wasn't asserted anywhere, and dictionary is the interesting negative — it wraps a value type but is deliberately not nested in either implementation, so pinning it means a later change can't quietly sweep it in.

Are these changes tested?

Yes. test_is_nested_or_struct fails on the current code and passes with the change.

I don't have a local C++ build, so I checked this by running the updated test_types.py against an installed pyarrow 25.0.0 with the same one-line change applied to its types.py. Before the change that file had 87 passing with test_is_nested_or_struct failing; after it, 88 passing. The two errors and one failure I see in both runs are environmental on my machine and unrelated — the errors are the pickle_module fixture, which comes from a conftest I wasn't loading, and the failure is test_pytz_timezone_roundtrip.

Are there any user-facing changes?

Yes, though it's small. pa.types.is_nested() now returns True for run-end encoded types where it previously returned False. Anything branching on that predicate will take the nested path for these types, which is the intended answer and what the C++ implementation has always given. Nothing inside pyarrow reads is_nested or _NESTED_TYPES, so the effect is limited to callers.

pyarrow.types.is_nested() answered False for run-end encoded types while
the C++ arrow::is_nested() answers True for them. The Python predicate
reads from a hardcoded _NESTED_TYPES set that never had
Type_RUN_END_ENCODED added to it; run-end encoded was the only type the
two definitions still disagreed about.

Also extend test_is_nested_or_struct with map and dictionary, so the set
is pinned at both ends rather than only for the types that are nested.
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #50847 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant