Skip to content

parquet page level skipping (page index pruning) doesn't work with evolved schemas #5104

Description

@alamb

Describe the bug
parquet page level skipping (page index pruning) panic's with evolved schemas

To Reproduce

Enable setting like

        pub enable_page_index: bool, default = true

And then run test:

cargo test -p datafusion --lib -- evolved_schema_disjoint_schema_filter

Which fails

test physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter ... FAILED

failures:

---- physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter' panicked at 'index out of bounds: the len is 1 but the index is 1', /Users/alamb/.cargo/registry/src/github.com-1ecc6299db9ec823/parquet-31.0.0/src/file/metadata.rs:251:10

Expected behavior
Test should pass

Additional context
I found this while working to enable this feature by default
#5099

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions