[Parquet] Adaptive Parquet Predicate Pushdown - #8733
Merged
Conversation
Contributor
Author
|
The version without synthetic page: |
This was referenced Oct 29, 2025
Contributor
|
😮 thank you @hhhizzz -- I plan to review this PR carefully, but it will likely take me a few days |
Contributor
|
fyi @zhuqi-lucas and @XiangpengHao |
Contributor
|
🤖 |
zhuqi-lucas
reviewed
Oct 30, 2025
Contributor
|
🤖: Benchmark completed Details
|
Contributor
|
🤖 |
Contributor
|
🤖: Benchmark completed Details
|
Contributor
|
🤖 |
Contributor
|
🤖: Benchmark completed Details
|
Contributor
Author
|
For how I get the the average length to use the mask, here's some statistic, you can checkout to (https://github.com/hhhizzz/arrow-rs/tree/rowselectionempty-charts) and run One column
|
alamb
reviewed
Oct 30, 2025
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
First of all, thank you so much @hhhizzz -- I think this is really nice change and the code is well structured and a pleasure to read. Also thank you to @zhuqi-lucas for setting the stage for much of this work
Given the performance results so far (basically as good or better as the existing code) I think this PR is almost ready to go
The only thing I am not sure about is the null page / skipping thing -- I left more comments inline
I think there are several additional improvements that could be done as follow on work:
- The heuristic for when to use the masking strategy can likely be improved based on the types of values being filtered (for example the number of columns or the inclusion of StringView)
- Avoid creating
RowSelectionjust to turn it back to a BooleanArray (I left comments inline)
alamb
reviewed
Oct 30, 2025
alamb
reviewed
Oct 30, 2025
13 tasks
tustvold
reviewed
Nov 1, 2025
tustvold
reviewed
Nov 1, 2025
tustvold
reviewed
Nov 1, 2025
tustvold
reviewed
Nov 1, 2025
Contributor
There was a problem hiding this comment.
I took a quick look, whilst I think orchestrating this skipping at the RecordReader level does have a certain elegance, it runs into the issue that the masked selections aren't necessarily page-aligned.
By definition the mask selection strategy requests rows that weren't part of the original selection, the problem is that this could result in requesting rows for pages that we know are irrelevant. In some cases this just results in wasted IO, however, when using prefetching IO systems (such as AsyncParquetReader) this results in errors. The hack of creating empty pages I'm not a big fan of.
I think a better solution would be to ensure we only construct MaskChunk that don't cross page boundaries. Ideally this would be done on a per-leaf column basis, but tbh I suspect just doing it globally would probably work just fine.
Edit: If one was feeling fancy, one could ignore page boundaries where both pages were present in the original selection, although in practice I suspect this not to make a huge difference.
tustvold
reviewed
Nov 1, 2025
hhhizzz
marked this pull request as draft
November 3, 2025 09:53
hhhizzz
force-pushed
the
rowselectionempty
branch
from
November 4, 2025 12:22
ad51d87 to
ed51620
Compare
53 tasks
Contributor
|
I plan to leave this PR open for a few more days to gather any additional comments and then merge it in I may also do some prototyping of additional optimizations (avoiding the masks and fetches entirely) for fun Thank you again @hhhizzz -- I think this PR will finally get us past the hurdle for making predicate pushdown work well for all cases |
This was referenced Nov 14, 2025
Contributor
|
Let's gogogogogogo! I am working on filing follow on tickets too |
Contributor
|
Thank you again @hhhizzz , @zhuqi-lucas and @tustvold |
This was referenced Nov 14, 2025
Contributor
Contributor
Author
|
@alamb @zhuqi-lucas @tustvold Thank you — I know this might be a bit off the main topic, but I still want to express my appreciation. We still have a few follow-up tickets related to this improvement, and I’ll continue exploring them. Thanks again for your patience and support. |
Contributor
It has been my pleasure reviewing your code and interacting with you in the context of this PR. It is a really nice contribution and I think (hopefully) has finally broken us through the barrier needed to get this feature rolled out to everyone. 🙏 I look forward to continue working together |
github-merge-queue Bot
pushed a commit
to apache/datafusion
that referenced
this pull request
Dec 3, 2025
…prior to parquet 57.1.0 upgrade (#19003) ~Draft until #18820 is merged~ ## Which issue does this PR close? - Follow on to #18820 ## Rationale for this change The parquet 57.1.0 upgrade includes a new adaptive filter from @hhhizzz : - apache/arrow-rs#8733 Our testing shows this is faster in all cases, but I want to have an escape valve for people to turn it off if they hit some issue. I had originally included this in #18820 but @rluvaton suggested it would be easier to understand as its own PR in #18820 (review) ## What changes are included in this PR? 1. Add a `force_filter_selections` config setting 2. Add configuration guide 3. Add tests ## Are these changes tested? Yes ## Are there any user-facing changes? A new boolean flag
1 task
6 tasks
alamb
pushed a commit
that referenced
this pull request
Feb 3, 2026
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Part of #9269. # Rationale for this change Check issue <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? Moved tests created in #8733 to a separate module -> `parquet/tests/arrow_reader/row_filter.rs` <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? No need, code movement. <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> # Are there any user-facing changes? No, code movement. <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
6 tasks
alanprot
added a commit
to alanprot/arrow-rs
that referenced
this pull request
Aug 11, 2026
`ParquetRecordBatchReader::try_new_with_row_groups` builds its read plan with `RowSelectionPolicy::default()` (`Auto`) and no loaded row ranges. For a fragmented selection `Auto` resolves to mask execution, and `MaskCursor` then decodes every row a chunk covers before applying the mask. A caller whose `RowGroups` holds only the pages its `RowSelection` touches - the pattern `RowSelection::scan_ranges` exists to support - therefore fails with "Invalid offset in sparse column chunk data". `ParquetPushDecoder` is unaffected because `RowGroupReaderBuilder` calls `prepare_selection_for_page_skipping`, but that is not reachable from this constructor. The regression dates to apache#8733, which introduced the mask strategy and the guard together; this constructor was never wired into it and has not been touched since apache#4307. Add `RowGroups::offset_index`, defaulting to `None`, for implementations to surface the offset index they pruned with, and derive the loaded row ranges from it inside `try_new_with_row_groups`. The derivation is conservative: a caller that loaded more pages than the selection touches still gets a subset of what it actually loaded, so confining mask execution to those ranges is always safe. `RowSelection::loaded_row_ranges` holds the derivation, and `loaded_row_ranges_for_projection` in the push decoder now delegates to it. No public types are added and no signatures change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Predicate pushdown and row filters often yield very short alternating select/skip runs; materialising those runs through a
VecDeque<RowSelector>is both allocation-heavy and prone to panicswhen entire pages are skipped and never decoded. This PR introduces a mask-backed execution path for short selectors, adds the heuristics and guards needed to decide between masks and selectors,
and provides tooling to measure the trade‑offs so we can tune the threshold confidently.
What changes are included in this PR?
RowSelectionStrategyplus aRowSelectionCursorthat can iterate either a boolean mask or the legacy selector queue, along with a public guard/override so tests and benchmarks cantweak the average-selector-length heuristic.
ReadPlanBuildernow trims selections, estimates their density, and chooses the faster mask strategy when safe;ParquetRecordBatchReaderstreams mask chunks, skips contiguous gaps, filtersthe projected batch with Arrow’s boolean kernel, and still falls back to selectors when needed.
RowSelectioncan now inspect offset indexes to detect when a selection would skip entire pages; both the synchronous and asynchronous readers consult that signal so row filters no longerpanic when predicate pruning drops whole pages.
row_selection_stateCriterion benchmark plusdev/row_selection_analysis.pyto run the bench, export CSV summaries, and render comparative plots across selector lengths, columnwidths, and Utf8View payload sizes; wired the bench into
parquet/Cargo.toml.test_row_selection_interleaved_skip,test_row_selection_mask_sparse_rows,test_row_filter_full_page_skip_is_handledand itsasync twin) and updated the push-decoder size assertion to reflect the new state.
Are these changes tested?
ReadPlanBuilderthreshold tests; the Criterion bench + Python tooling provide manual validation for performancetuning. Full parquet/arrow test suites will still run in CI.
Are there any user-facing changes?
RowSelectionStrategy,RowSelectionCursor, andset_avg_selector_len_mask_thresholdfor experimentation, and developers gain the new benchmarking/plottingworkflow. No breaking API changes were introduced.