feat(pruning): expose pruning predicate IN-list rewrite size cap as a config option - #24074
Merged
zhuqi-lucas merged 8 commits intoAug 6, 2026
Conversation
Issue: apache#24059 `PruningPredicate` rewrites `col IN (v1..vn)` into a chain of per-value min/max checks (via `build_predicate_expression`), but only when `n` is below a hardcoded `MAX_LIST_VALUE_SIZE_REWRITE = 20`. Beyond that, the IN branch falls through to `unhandled_hook`, which by default returns `TRUE` — so row-group / file-range statistics pruning does not fire at all for IN lists longer than 20. This is problematic for query patterns that pass a batch of identifiers as `col IN (...)` (REST endpoints filtering by a page of ~25-100 values, ORM-generated `WHERE id IN (25 items)` queries, batched crawlers). On a table sorted by `col`, the reader is forced to materialize the filter column across every row group instead of skipping row groups whose stats disagree with the IN set. ## Changes - Add `datafusion.execution.parquet.pruning_max_in_list_size: usize` (default `20`, preserving existing behaviour) to `TableParquetOptions` next to `max_predicate_cache_size`. - Add `PredicateRewriter::with_max_in_list_size(usize) -> Self` builder, mirroring the existing `with_unhandled_hook`. - Add `PruningPredicate::try_new_with_max_in_list_size` variant. - Add `build_pruning_predicate_with_max_in_list_size` variant of the public helper. - Make `MAX_LIST_VALUE_SIZE_REWRITE` `pub const` so callers can reference the historical default explicitly. - Wire the value through `datasource-parquet`: - `ParquetSource::pruning_max_in_list_size()` reads from `TableParquetOptions.global`. - `ParquetMorselizer` / `PreparedParquetOpen` / `RowGroupPruner` carry the value alongside `max_predicate_cache_size`. - `build_pruning_predicates` (opener) accepts the size and forwards to `build_pruning_predicate_with_max_in_list_size`. ## Backward compatibility - Public `PruningPredicate::try_new` and `build_pruning_predicate` are preserved as thin wrappers passing the historical default. - Internal `build_predicate_expression` takes a new `usize` parameter (crate-private). - Default value of the config option is `20`, so behaviour is unchanged unless the option is set explicitly. ## Tests - `row_group_predicate_in_list_rewritten_at_raised_cap`: `PredicateRewriter::with_max_in_list_size(32)` rewrites a 25-item IN into per-value min/max checks instead of falling through to `true`. - `row_group_predicate_in_list_disabled_at_zero_cap`: cap = 0 skips the IN rewrite even for small lists (opt-out path). - Existing `row_group_predicate_in_list_to_many_values` continues to pass, guarding the default-20 behaviour.
github-actions
Bot
added
common
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes the IN (...)-list rewrite cap used by PruningPredicate configurable (instead of a hardcoded 20), and plumbs that setting through the parquet datasource so row-group / file-range stats pruning can remain effective for larger IN lists when users opt in.
Changes:
- Add
datafusion.execution.parquet.pruning_max_in_list_size(default20) to parquet execution config and thread it throughParquetSource→ opener/morselizer → row-group pruner. - Expose the historical default as
pub const MAX_LIST_VALUE_SIZE_REWRITEand add API variants/builders to pass an explicit cap (with_max_in_list_size,try_new_with_max_in_list_size,build_pruning_predicate_with_max_in_list_size). - Add unit tests covering raised-cap rewrite behavior and cap=0 opt-out behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| datafusion/pruning/src/pruning_predicate.rs | Adds configurable IN-list rewrite cap to predicate rewriting/pruning APIs and tests it. |
| datafusion/pruning/src/lib.rs | Re-exports the new const and helper function as part of the public pruning API. |
| datafusion/datasource-parquet/src/source.rs | Reads the new config from TableParquetOptions.global and propagates it into pruning predicate construction. |
| datafusion/datasource-parquet/src/push_decoder.rs | Stores and applies the cap when (re)building pruning predicates in RowGroupPruner. |
| datafusion/datasource-parquet/src/opener/mod.rs | Threads the cap through ParquetMorselizer/PreparedParquetOpen and uses the new helper to build pruning predicates. |
| datafusion/common/src/file_options/parquet_writer.rs | Updates writer options destructuring to account for the newly added parquet option field. |
| datafusion/common/src/config.rs | Introduces the pruning_max_in_list_size parquet execution config option with documentation. |
Suppressed comments (3)
datafusion/pruning/src/pruning_predicate.rs:493
- Docs reference
datafusion.execution.pruning_max_in_list_size, but the actual config option isdatafusion.execution.parquet.pruning_max_in_list_size. Update this reference so callers can find the right setting.
/// Same as [`PruningPredicate::try_new`] but with an explicit cap on the
/// size of `IN (...)` lists rewritten into per-value statistics checks.
/// Query engines typically pass
/// `datafusion.execution.pruning_max_in_list_size` here.
datafusion/pruning/src/pruning_predicate.rs:1404
- This doc comment points to
datafusion.execution.pruning_max_in_list_size, but the new option is namespaced underparquet(datafusion.execution.parquet.pruning_max_in_list_size). Fixing the key avoids confusion for users trying to set the default cap explicitly.
/// Default maximum number of entries in an `IN (...)` list that will be
/// rewritten into a chain of per-value min/max checks by
/// [`build_predicate_expression`]. Callers threading a [`PredicateRewriter`]
/// can override this via [`PredicateRewriter::with_max_in_list_size`], and
/// query engines can wire it from the
/// `datafusion.execution.pruning_max_in_list_size` config option.
pub const MAX_LIST_VALUE_SIZE_REWRITE: usize = 20;
datafusion/pruning/src/pruning_predicate.rs:1445
- This builder method's docs reference
datafusion.execution.pruning_max_in_list_size, but the config key isdatafusion.execution.parquet.pruning_max_in_list_size. Update the docs to match the actual option name.
/// The default (see [`MAX_LIST_VALUE_SIZE_REWRITE`]) preserves the
/// historical behaviour. Callers wiring config through can override via
/// `datafusion.execution.pruning_max_in_list_size`.
pub fn with_max_in_list_size(mut self, max_in_list_size: usize) -> Self {
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+399
to
+402
| /// Same as [`build_pruning_predicate`] but with an explicit cap on the size | ||
| /// of `IN (...)` lists rewritten into per-value statistics checks. Query | ||
| /// engines typically pass `datafusion.execution.pruning_max_in_list_size` | ||
| /// here. |
Contributor
Author
There was a problem hiding this comment.
Fixed by renaming to datafusion.execution.parquet.max_in_list_size (per alamb below) — docs now match the actual config key.
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
github-actions
Bot
added
auto detected api change
The new ParquetOptions::pruning_max_in_list_size field was not carried through the proto layer, so every explicit ParquetOptions initializer in datafusion-proto-common and datafusion-proto failed to compile (E0063), which also cascaded into the MSRV job. - Add uint64 pruning_max_in_list_size = 38 to the ParquetOptions proto message and regenerate proto-common (prost.rs, pbjson.rs) and proto-models (datafusion_proto_common.rs). - Map the field in proto-common from_proto/to_proto and in the proto crate's file_formats TryFromProto/IntoProto for TableParquetOptions. - Reorder the datafusion_pruning import (cargo fmt) and drop the now unused build_pruning_predicate import.
zhuqi-lucas
force-pushed
the
qizhu/config-pruning-in-list-rewrite-size
branch
from
August 4, 2026 06:32
038b3ee to
5f4cfa1
Compare
- cargo doc: the public MAX_LIST_VALUE_SIZE_REWRITE doc linked the private build_predicate_expression via an intra-doc link; demote it to a code span and correct the config path to datafusion.execution.parquet.pruning_max_in_list_size. - Regenerate configs.md for the new pruning_max_in_list_size option. - Add the two pruning_max_in_list_size rows to information_schema.slt (SHOW ALL and the df_settings description listing).
github-actions
Bot
added
documentation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24074 +/- ##
========================================
Coverage 80.98% 80.98%
========================================
Files 1104 1104
Lines 378826 378968 +142
Branches 378826 378968 +142
========================================
+ Hits 306797 306912 +115
- Misses 53813 53832 +19
- Partials 18216 18224 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
alamb
reviewed
Aug 4, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Looks good @zhuqi-lucas . I had some suggestions on API design and naming, but the overall idea makes a lot of sense
| enable_row_group_stats_pruning: false, | ||
| coerce_int96: None, | ||
| max_predicate_cache_size: None, | ||
| pruning_max_in_list_size: MAX_LIST_VALUE_SIZE_REWRITE, |
Contributor
There was a problem hiding this comment.
it is strange to me that these names are not the same -- I would expect something like
pruning_max_in_list_size: PRUNING_MAX_IN_LIST_SIZE,
Contributor
Author
There was a problem hiding this comment.
Fixed — renamed the field to max_in_list_size and the const to MAX_IN_LIST_SIZE so the shapes match.
| predicate, | ||
| file_schema, | ||
| predicate_creation_errors, | ||
| MAX_LIST_VALUE_SIZE_REWRITE, |
Contributor
There was a problem hiding this comment.
Why not just add the parameter to build_pruning_predicate ?
If we are going to introduce a new API, perhaps we can make one that is more future proof, like a builder
let pruning_predicate = PruningPredicaateBuilder::new()
.with_file_schema(file_schema)
.with_error_counter(predicate_creation_errors)
.build(predicate)?;That way if we add new parameters we have a place to put them
Contributor
Author
There was a problem hiding this comment.
Great suggestion — implemented as PruningPredicateBuilder with .with_file_schema(...), .with_error_counter(...), .with_max_in_list_size(...), and .build(predicate) returning Option<Arc<PruningPredicate>> for the parquet scan path, plus .try_build(predicate) returning Result<PruningPredicate> for callers that want to surface errors themselves. The standalone build_pruning_predicate_with_max_in_list_size and PruningPredicate::try_new_with_max_in_list_size are gone.
| /// before calling this method to make sure the expressions can be used for pruning. | ||
| pub fn try_new(mut expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { | ||
| pub fn try_new(expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { | ||
| Self::try_new_with_max_in_list_size(expr, schema, MAX_LIST_VALUE_SIZE_REWRITE) |
Contributor
There was a problem hiding this comment.
Same comment above related to simplifying this API via a builder rather than more methods
Contributor
Author
There was a problem hiding this comment.
Removed the try_new_with_max_in_list_size variant. PruningPredicate::try_new keeps its historical signature; the new PruningPredicateBuilder is the entry point for callers that want to override max_in_list_size (or supply an error counter).
| /// container. Set to 0 to disable the rewrite path entirely. | ||
| /// | ||
| /// The default of 20 preserves the previous hardcoded behaviour. | ||
| pub pruning_max_in_list_size: usize, default = 20 |
Contributor
There was a problem hiding this comment.
Also I suggest changing this to be something more conisstent with the others names like max_predicate_cache_size
Perhaps something likemax_in_list_size or max_in_list_pruning_size
Contributor
Author
There was a problem hiding this comment.
Renamed to max_in_list_size (matches the max_predicate_cache_size neighbour). Section is already execution.parquet.* so the pruning context is inferable from the key path.
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Contributor
Author
|
Thanks for the review and good suggestions @alamb, addressed comments now. |
- Introduce PruningPredicateBuilder per alamb's suggestion, replacing the ad-hoc try_new_with_max_in_list_size / build_pruning_predicate_with_max_in_list_size helpers with a single builder that also carries the error counter. - Rename config option pruning_max_in_list_size -> max_in_list_size to match the max_predicate_cache_size naming style (per alamb). - Rename const MAX_LIST_VALUE_SIZE_REWRITE -> MAX_IN_LIST_SIZE so the field name and the const name are consistent (per alamb's opener/mod.rs comment). - Trim the config-option doc (removed accidentally-duplicated 'of' word, tightened wording). - Regenerate docs/source/user-guide/configs.md and datafusion/sqllogictest/test_files/information_schema.slt. - Add PruningPredicateBuilder unit test verifying max_in_list_size is threaded end-to-end (default -> 'true'; raised cap -> real statistics predicate).
zhuqi-lucas
force-pushed
the
qizhu/config-pruning-in-list-rewrite-size
branch
from
August 5, 2026 12:07
92b7e69 to
ab65428
Compare
alamb
approved these changes
Aug 5, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Look great to me -- thank you @zhuqi-lucas
| file_schema, | ||
| predicate_creation_errors, | ||
| ) | ||
| PruningPredicateBuilder::new() |
| .build(predicate) | ||
| } | ||
|
|
||
| /// Builder for a [`PruningPredicate`]. Groups optional configuration — |
| /// It is recommended that you pass the expressions through [`PhysicalExprSimplifier`] | ||
| /// before calling this method to make sure the expressions can be used for pruning. | ||
| pub fn try_new(mut expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { | ||
| pub fn try_new(expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { |
Contributor
There was a problem hiding this comment.
Maybe as a follow on PR we want to direct people to PruningPredicateBuilder 🤔 in comments / deprecate the try_new and move this construction into the builder
Contributor
Author
There was a problem hiding this comment.
Good idea — filed #24128 to track the try_new deprecation + builder migration as a follow-up.
alamb
changed the title
Contributor
|
I update an output file in 8ad8ad8 and merged up from main |
Contributor
Author
|
Thanks @alamb for review and good suggestions, merged now. |
zhuqi-lucas
added a commit
to massive-com/arrow-datafusion
that referenced
this pull request
Aug 6, 2026
Backport of apache#24074 to our internal branch-54 fork. Adds `datafusion.execution.parquet.max_in_list_size` (default 20, matching the previous hardcoded behaviour). Raising this lifts the cap under which IN (...) predicates are rewritten into per-value min/max checks, so row-group / file-range statistics pruning stays effective for larger IN lists (typical REST-endpoint pattern of ~25-100 identifiers). ## Changes - Add `max_in_list_size: usize` (default 20) to `TableParquetOptions.global`. - Promote `MAX_LIST_VALUE_SIZE_REWRITE` -> `MAX_IN_LIST_SIZE` and make it `pub const`. - Introduce `PruningPredicateBuilder` (`.with_file_schema`, `.with_error_counter`, `.with_max_in_list_size`, `.build` / `.try_build`) replacing the ad-hoc `_with_max_in_list_size` variants. - Add `PredicateRewriter::with_max_in_list_size` builder method. - Wire the config from `ParquetSource::max_in_list_size()` through `ParquetMorselizer` -> `PreparedParquetOpen` -> `build_pruning_predicates`. - Regenerate proto (field 37) + `configs.md` + surgical `information_schema.slt` patch. ## Tests - `row_group_predicate_in_list_rewritten_at_raised_cap` - `PredicateRewriter::with_max_in_list_size(32)` rewrites 25-item IN into per-value min/max chain instead of falling through to `true`. - `row_group_predicate_in_list_disabled_at_zero_cap` - cap=0 opt-out. - `pruning_predicate_builder_threads_max_in_list_size` - end-to-end builder API threading. Upstream: apache#24074
zhuqi-lucas
added a commit
to massive-com/arrow-datafusion
that referenced
this pull request
Aug 6, 2026
Backport of apache#24074 to our internal branch-54 fork. Adds `datafusion.execution.parquet.max_in_list_size` (default 20, matching the previous hardcoded behaviour). Raising this lifts the cap under which IN (...) predicates are rewritten into per-value min/max checks, so row-group / file-range statistics pruning stays effective for larger IN lists (typical REST-endpoint pattern of ~25-100 identifiers). ## Changes - Add `max_in_list_size: usize` (default 20) to `TableParquetOptions.global`. - Promote `MAX_LIST_VALUE_SIZE_REWRITE` -> `MAX_IN_LIST_SIZE` and make it `pub const`. - Introduce `PruningPredicateBuilder` (`.with_file_schema`, `.with_error_counter`, `.with_max_in_list_size`, `.build` / `.try_build`) replacing the ad-hoc `_with_max_in_list_size` variants. - Add `PredicateRewriter::with_max_in_list_size` builder method. - Wire the config from `ParquetSource::max_in_list_size()` through `ParquetMorselizer` -> `PreparedParquetOpen` -> `build_pruning_predicates`. - Regenerate proto (field 37) + `configs.md` + surgical `information_schema.slt` patch. ## Tests - `row_group_predicate_in_list_rewritten_at_raised_cap` - `PredicateRewriter::with_max_in_list_size(32)` rewrites 25-item IN into per-value min/max chain instead of falling through to `true`. - `row_group_predicate_in_list_disabled_at_zero_cap` - cap=0 opt-out. - `pruning_predicate_builder_threads_max_in_list_size` - end-to-end builder API threading. Upstream: apache#24074
zhuqi-lucas
added a commit
to massive-com/arrow-datafusion
that referenced
this pull request
Aug 6, 2026
Backport of apache#24074 to our internal branch-54 fork. Adds `datafusion.execution.parquet.max_in_list_size` (default 20, matching the previous hardcoded behaviour). Raising this lifts the cap under which IN (...) predicates are rewritten into per-value min/max checks, so row-group / file-range statistics pruning stays effective for larger IN lists (typical REST-endpoint pattern of ~25-100 identifiers). ## Changes - Add `max_in_list_size: usize` (default 20) to `TableParquetOptions.global`. - Promote `MAX_LIST_VALUE_SIZE_REWRITE` -> `MAX_IN_LIST_SIZE` and make it `pub const`. - Introduce `PruningPredicateBuilder` (`.with_file_schema`, `.with_error_counter`, `.with_max_in_list_size`, `.build` / `.try_build`) replacing the ad-hoc `_with_max_in_list_size` variants. - Add `PredicateRewriter::with_max_in_list_size` builder method. - Wire the config from `ParquetSource::max_in_list_size()` through `ParquetMorselizer` -> `PreparedParquetOpen` -> `build_pruning_predicates`. - Regenerate proto (field 37) + `configs.md` + surgical `information_schema.slt` patch. ## Tests - `row_group_predicate_in_list_rewritten_at_raised_cap` - `PredicateRewriter::with_max_in_list_size(32)` rewrites 25-item IN into per-value min/max chain instead of falling through to `true`. - `row_group_predicate_in_list_disabled_at_zero_cap` - cap=0 opt-out. - `pruning_predicate_builder_threads_max_in_list_size` - end-to-end builder API threading. Upstream: apache#24074
zhuqi-lucas
added a commit
to massive-com/arrow-datafusion
that referenced
this pull request
Aug 10, 2026
…#70) Backport of apache#24074 to our internal branch-54 fork. Adds `datafusion.execution.parquet.max_in_list_size` (default 20, matching the previous hardcoded behaviour). Raising this lifts the cap under which IN (...) predicates are rewritten into per-value min/max checks, so row-group / file-range statistics pruning stays effective for larger IN lists (typical REST-endpoint pattern of ~25-100 identifiers). ## Changes - Add `max_in_list_size: usize` (default 20) to `TableParquetOptions.global`. - Promote `MAX_LIST_VALUE_SIZE_REWRITE` -> `MAX_IN_LIST_SIZE` and make it `pub const`. - Introduce `PruningPredicateBuilder` (`.with_file_schema`, `.with_error_counter`, `.with_max_in_list_size`, `.build` / `.try_build`) replacing the ad-hoc `_with_max_in_list_size` variants. - Add `PredicateRewriter::with_max_in_list_size` builder method. - Wire the config from `ParquetSource::max_in_list_size()` through `ParquetMorselizer` -> `PreparedParquetOpen` -> `build_pruning_predicates`. - Regenerate proto (field 37) + `configs.md` + surgical `information_schema.slt` patch. ## Tests - `row_group_predicate_in_list_rewritten_at_raised_cap` - `PredicateRewriter::with_max_in_list_size(32)` rewrites 25-item IN into per-value min/max chain instead of falling through to `true`. - `row_group_predicate_in_list_disabled_at_zero_cap` - cap=0 opt-out. - `pruning_predicate_builder_threads_max_in_list_size` - end-to-end builder API threading. Upstream: apache#24074
kosiew
pushed a commit
to kosiew/datafusion
that referenced
this pull request
Aug 12, 2026
… config option (apache#24074) ## Which issue does this PR close? - Closes apache#24059. ## Rationale `PruningPredicate` rewrites `col IN (v1..vn)` into a chain of per-value min/max checks (via `build_predicate_expression`), but only when `n <= MAX_LIST_VALUE_SIZE_REWRITE` — currently a hardcoded `20`. Beyond that, the IN branch falls through to `unhandled_hook`, which by default returns `TRUE`, so row-group and file-range statistics pruning does not fire at all for IN lists longer than 20. This is problematic for query patterns that pass a batch of identifiers as `col IN (...)` — REST endpoints filtering by a page of ~25-100 values, ORM-generated `WHERE id IN (25 items)` queries, batched crawlers. On a table sorted by `col`, the reader is forced to materialize the filter column across every row group instead of skipping row groups whose stats disagree with the IN set. Full context in apache#24059. ## What changes are included in this PR? - New config option `datafusion.execution.parquet.pruning_max_in_list_size: usize` (default `20`, preserving existing behaviour), placed next to `max_predicate_cache_size` on `TableParquetOptions.global`. - `MAX_LIST_VALUE_SIZE_REWRITE` promoted to `pub const` so callers can reference the historical default explicitly. - `PredicateRewriter::with_max_in_list_size(usize) -> Self` builder, mirroring the existing `with_unhandled_hook`. - `PruningPredicate::try_new_with_max_in_list_size` variant. - `build_pruning_predicate_with_max_in_list_size` variant of the public helper. - Value threaded through `datasource-parquet`: `ParquetSource::pruning_max_in_list_size()` reads from `TableParquetOptions.global`, propagates through `ParquetMorselizer` → `PreparedParquetOpen` → `RowGroupPruner`, then flows into `build_pruning_predicates` at the opener and `build_pruning_predicate_with_max_in_list_size` inside the dynamic row-group pruner. Internal `build_predicate_expression` gains a new `usize` parameter (crate-private). ## Backward compatibility - `PruningPredicate::try_new` and `build_pruning_predicate` are preserved as thin wrappers that pass the historical `MAX_LIST_VALUE_SIZE_REWRITE` default. All existing callers continue to work with unchanged behaviour. - The config option default is `20`, so behaviour is unchanged unless the option is set explicitly. ## Are these changes tested? Two new unit tests in `datafusion-pruning`: - `row_group_predicate_in_list_rewritten_at_raised_cap`: `PredicateRewriter::with_max_in_list_size(32)` rewrites a 25-item IN into per-value min/max checks OR'd together, instead of falling through to `true`. - `row_group_predicate_in_list_disabled_at_zero_cap`: `cap = 0` skips the IN rewrite even for small lists (opt-out path). The existing `row_group_predicate_in_list_to_many_values` continues to pass, guarding the default-20 behaviour. ## Are there any user-facing changes? Yes — one new config option (`datafusion.execution.parquet.pruning_max_in_list_size`, default `20`). Users who want row-group / file-range pruning for IN lists longer than 20 items can raise it (e.g., `SET datafusion.execution.parquet.pruning_max_in_list_size = 128`). New public API on `datafusion-pruning`: - `MAX_LIST_VALUE_SIZE_REWRITE: usize` (re-exported) - `PredicateRewriter::with_max_in_list_size(usize) -> Self` - `PruningPredicate::try_new_with_max_in_list_size(expr, schema, size)` - `build_pruning_predicate_with_max_in_list_size(predicate, schema, errors, size)` --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
rkrishn7
pushed a commit
to rkrishn7/datafusion
that referenced
this pull request
Aug 27, 2026
## Why are the changes needed? ### Which issue does this PR close? Related to apache#8668 and apache#8609; follows apache#24074 and the merged statistics-ordering fix apache#24525. ### Rationale for this change Queries often select a sparse set of string identifiers. Parquet min/max statistics can make these queries much cheaper by ruling out row groups or pages that cannot contain any requested identifier. For example, this query asks for 21 IDs, spaced ten apart: ```sql SET datafusion.execution.parquet.max_in_list_size = 1024; SELECT * FROM events WHERE customer_id IN ( 'id000', 'id010', 'id020', 'id030', 'id040', 'id050', 'id060', 'id070', 'id080', 'id090', 'id100', 'id110', 'id120', 'id130', 'id140', 'id150', 'id160', 'id170', 'id180', 'id190', 'id200' ); ``` A row group whose values fall between `id003` and `id007` cannot contain a match. The default pruning limit is 20, so this list is not eligible for the `IN` min/max rewrite unless the caller raises the limit. apache#24074 made that limit configurable. With a raised limit, DataFusion can already reject this row group, but it does so by constructing a growing expression tree resembling: ```text (min <= id000 AND id000 <= max) OR (min <= id010 AND id010 <= max) OR ... ``` For hundreds or thousands of identifiers, building and evaluating that tree can become expensive in its own right. Replacing the list with one enclosing range, `[id000, id200]`, would be cheaper, but would lose the gaps: that broad range overlaps `[id003, id007]` even though none of the requested IDs is present there. The aim is to keep the useful pruning precision of the existing per-value checks while making large lists cheaper to represent and evaluate. In the included local microbenchmark, evaluating 1,024 values against 4,096 intervals falls from 68.8 ms to 0.198 ms. This measures pruning work only, not end-to-end query speedup. ## What changes were proposed in this PR? ### What changes are included in this PR? Eligible large string lists are stored as a sorted, deduplicated set of values inside the pruning predicate, rather than expanded into one comparison branch per value. For each inclusive statistics interval, DataFusion finds the first requested value at or after the interval's minimum, then checks whether that value is also at or before its maximum. In the example, the first requested ID at or after `id003` is `id010`. Since `id010 > id007`, the row group can be skipped. An interval such as `[id019, id021]` must be kept because it could contain `id020`. This takes a binary search per interval after sorting the values once per constructed predicate, and the expression tree no longer grows with the number of IDs. The result remains a conservative pruning decision. An overlapping interval means only that a match is possible; the original `IN` expression still performs exact row filtering. The original literal information also remains available to other pruning mechanisms, including Bloom filters. Inverted or unusable bounds yield UNKNOWN. A single known bound can still exclude the entire domain; otherwise, incomplete bounds yield UNKNOWN and keep the container eligible for reading. **Limits and scope** For a nonempty input list of `N` values, before this optimization deduplicates it, and configured cap `C = datafusion.execution.parquet.max_in_list_size`: | Condition | Min/max pruning behavior | | --- | --- | | `N <= min(20, C)` | Existing per-value rewrite | | `20 < N <= C`, positive, non-null literal strings on a direct string column | Compact sorted-domain lookup | | `20 < N <= C`, other lists | Existing per-value rewrite, including non-string lists, `NOT IN`, and NULL-containing lists | | `N > C` | This `IN` min/max rewrite is skipped; it does not disable all pruning | The compact path supports `Utf8`, `LargeUtf8`, `Utf8View`, and dictionary-encoded string columns. Unsupported expressions keep their existing handling. The default cap remains 20, and `C = 0` disables this `IN` min/max rewrite. Other predicates and literal/containment pruning, including Bloom filters, remain available. Raising the cap can still produce expensive comparison trees for lists that do not use the compact path. The lower threshold of 20 preserves the existing expression shape and limits this PR's scope; it is not a measured performance crossover. Page-index pruning now receives the same configured cap as row-group pruning, so raising the limit can benefit both. An interval search requires bounds in the same comparison order as the query. The compact path reads the same statistics adapters as the existing per-value path. The already-merged apache#24525 masks untrusted Parquet byte-array bounds. For other statistics sources, `PruningStatistics` requires conservative bounds in Arrow's comparison order; arbitrary provider bounds are trusted rather than validated by this expression. ### Are there any user-facing changes? Users who raise `datafusion.execution.parquet.max_in_list_size` get cheaper min/max pruning for eligible large string lists, and page pruning now honors that setting. The configuration default, exact query results, and existing public APIs are unchanged. This is a focused optimization for literal string lists, not a general rewrite of every large `IN` expression. ## How was this PR tested? ### Are these changes tested? The pruning-crate suite passed 95 tests. The standalone Parquet regressions use lists of 20, 21, 256, and 1,024 values and check both exact query results and scan/pruning metrics. They cover gaps inside the list's enclosing range, row-group pruning, page-only pruning, and the default and zero-cap controls. Two correctness regressions exercise the less obvious interactions. A direct physical-source test uses `NOT IN (..., NULL)`, row-filter pushdown, and `LIMIT 1`, so logical optimizer folding cannot hide an incorrect decision to bypass filtering. A real-file test combines this PR with apache#24525 and verifies that compact page pruning cannot lose a matching row when the footer's ordering is missing or unknown. Against unchanged Apache `f1f0449a`, the positive row-group/page tests fail as expected; the `NOT IN (..., NULL)` control passes. The benchmark compares the actual raised-cap `IN` path on Apache `f1f0449a` and this patch, using separate build directories and checking that both return the same nontrivial pruning results. Local results on an Apple M5 Max (18 CPUs, 128 GiB), Rust 1.97.0, `release-nonlto`, 20 samples: | Values | Predicate construction, main → PR | Evaluate 4,096 intervals, main → PR | | --- | --- | --- | | 20 | 31.1 → 27.4 µs | 177.6 → 165.7 µs | | 21 | 30.5 → 3.52 µs | 183.8 → 98.1 µs | | 256 | 356.6 → 21.4 µs | 5.53 → 0.144 ms | | 1,024 | 1.466 → 0.081 ms | 68.8 → 0.198 ms | A balanced explicit OR tree is included as another comparison: at 1,024 values it takes 8.03 ms to evaluate the same intervals. These measurements isolate pruning overhead; no end-to-end workload improvement is claimed. The latest documentation/comment update passed configuration-doc generation, formatting, all-targets/all-features Clippy with warnings denied, and `./dev/rust_lint.sh` (including the workspace Rust documentation check). Focused validation also passed all 3 Parquet IN-list integration tests and all 9 Parquet statistics-ordering tests. The CI follow-up in [1a2b38c](apache@1a2b38c) updates the `SHOW ALL VERBOSE` snapshot to match the revised `max_in_list_size` help text. The stale row caused both the cargo test and benchmark verification jobs to fail. After the correction, the targeted `information_schema.slt` test and full extended suite passed: 10,796 Rust tests (8 ignored) and all 505 SQL logic files. Formatting, Clippy with all targets and features and warnings denied, and `./dev/rust_lint.sh` also passed. <details> <summary>Validation commands</summary> ```sh cargo test --locked --profile ci -p datafusion-pruning cargo test --locked --profile ci -p datafusion \ --test parquet_integration string_in_list_pruning cargo bench --locked --profile release-nonlto -p datafusion-pruning \ --bench string_in_list_pruning -- \ --sample-size 20 --warm-up-time 0.5 --measurement-time 1 --noplot RUST_BACKTRACE=1 cargo test --locked --profile ci \ --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli \ --workspace --lib --tests --bins \ --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption ``` </details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale
PruningPredicaterewritescol IN (v1..vn)into a chain of per-value min/max checks (viabuild_predicate_expression), but only whenn <= MAX_LIST_VALUE_SIZE_REWRITE— currently a hardcoded20. Beyond that, the IN branch falls through tounhandled_hook, which by default returnsTRUE, so row-group and file-range statistics pruning does not fire at all for IN lists longer than 20.This is problematic for query patterns that pass a batch of identifiers as
col IN (...)— REST endpoints filtering by a page of ~25-100 values, ORM-generatedWHERE id IN (25 items)queries, batched crawlers. On a table sorted bycol, the reader is forced to materialize the filter column across every row group instead of skipping row groups whose stats disagree with the IN set.Full context in #24059.
What changes are included in this PR?
datafusion.execution.parquet.pruning_max_in_list_size: usize(default20, preserving existing behaviour), placed next tomax_predicate_cache_sizeonTableParquetOptions.global.MAX_LIST_VALUE_SIZE_REWRITEpromoted topub constso callers can reference the historical default explicitly.PredicateRewriter::with_max_in_list_size(usize) -> Selfbuilder, mirroring the existingwith_unhandled_hook.PruningPredicate::try_new_with_max_in_list_sizevariant.build_pruning_predicate_with_max_in_list_sizevariant of the public helper.datasource-parquet:ParquetSource::pruning_max_in_list_size()reads fromTableParquetOptions.global, propagates throughParquetMorselizer→PreparedParquetOpen→RowGroupPruner, then flows intobuild_pruning_predicatesat the opener andbuild_pruning_predicate_with_max_in_list_sizeinside the dynamic row-group pruner.Internal
build_predicate_expressiongains a newusizeparameter (crate-private).Backward compatibility
PruningPredicate::try_newandbuild_pruning_predicateare preserved as thin wrappers that pass the historicalMAX_LIST_VALUE_SIZE_REWRITEdefault. All existing callers continue to work with unchanged behaviour.20, so behaviour is unchanged unless the option is set explicitly.Are these changes tested?
Two new unit tests in
datafusion-pruning:row_group_predicate_in_list_rewritten_at_raised_cap:PredicateRewriter::with_max_in_list_size(32)rewrites a 25-item IN into per-value min/max checks OR'd together, instead of falling through totrue.row_group_predicate_in_list_disabled_at_zero_cap:cap = 0skips the IN rewrite even for small lists (opt-out path).The existing
row_group_predicate_in_list_to_many_valuescontinues to pass, guarding the default-20 behaviour.Are there any user-facing changes?
Yes — one new config option (
datafusion.execution.parquet.pruning_max_in_list_size, default20). Users who want row-group / file-range pruning for IN lists longer than 20 items can raise it (e.g.,SET datafusion.execution.parquet.pruning_max_in_list_size = 128).New public API on
datafusion-pruning:MAX_LIST_VALUE_SIZE_REWRITE: usize(re-exported)PredicateRewriter::with_max_in_list_size(usize) -> SelfPruningPredicate::try_new_with_max_in_list_size(expr, schema, size)build_pruning_predicate_with_max_in_list_size(predicate, schema, errors, size)