Skip RowFilter and page pruning for fully matched row groups - #21637
Merged
xudong963 merged 15 commits intoMay 15, 2026
Conversation
xudong963
force-pushed
the
datafusion/issue-19028-benchmark
branch
from
April 15, 2026 05:36
54a4166 to
5da11ea
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
xudong963
marked this pull request as draft
April 15, 2026 05:45
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
xudong963
marked this pull request as ready for review
April 15, 2026 07:47
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
xudong963
force-pushed
the
datafusion/issue-19028-benchmark
branch
from
April 16, 2026 07:43
f0e02e9 to
d6c3879
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
|
🤔 the benchmarks look slower -- maybe we can profile some of those queries and find space to get the performance back |
This was referenced May 13, 2026
Closed
Member
Author
|
run benchmark clickbench_partitioned |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing datafusion/issue-19028-benchmark (d0b4c30) to 937dfda (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Member
Author
@alamb Good finding to avoid the PR introducing regression! I profiled the repeated ClickBench partitioned slow queries (
The issue was that I fixed this by removing the per-file Now the benchmark is good: #21637 (comment) |
Contributor
|
run benchmark clickbench_partitioned |
Contributor
Wild -- that seems like non trivial overhead Looking at the code and what you changed, maybe it is because the metric builder is expensive (it is copying strings) |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing datafusion/issue-19028-benchmark (426154e) to 937dfda (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
alamb
approved these changes
May 14, 2026
Contributor
There was a problem hiding this comment.
Thanks @xudong963 -- this is a really nice piece of engineering
| /// Record pages whose page-index pruning was skipped because the containing | ||
| /// row group was fully matched by row-group statistics. | ||
| /// | ||
| /// The counter is only registered when there is a non-zero value. This keeps |
Contributor
There was a problem hiding this comment.
I wonder if we should apply the same pattern to the other metrics (lazily initialize them) -- if you can get a few percent in this query maybe it would get us a few in the others
Member
Author
There was a problem hiding this comment.
This is a nice follow-up exploration, #22189 created an issue for this
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Member
Author
This was referenced May 15, 2026
zhuqi-lucas
added a commit
to zhuqi-lucas/arrow-datafusion
that referenced
this pull request
Aug 22, 2026
… slt churn) Follow the add_page_index_pages_skipped_by_fully_matched pattern from apache#21637: the counter is registered only when a suppression actually fires, via a RowFilterSkippedFullyMatchedMetric holder that keeps a live Count handle after first registration (the stream records suppressions as they happen, so a fire-once helper is not enough). - ParquetFileMetrics loses the eager pub field (no public API change, no semver flag). - 41 zero-valued row_filter_skipped_fully_matched=0 entries disappear from EXPLAIN ANALYZE baselines across push_down_filter_parquet.slt, explain_analyze.slt and dynamic_filter_pushdown_config.slt; the =1 assertions in dynamic_row_group_pruning.slt remain. Suggested by @adriangb in the apache#23696 decomposition review.
zzcclp
pushed a commit
to zzcclp/arrow-datafusion
that referenced
this pull request
Aug 25, 2026
…pache#23696) ## Which issue does this PR close? - Closes apache#23067. ## Rationale for this change When Parquet statistics prove that **every** row of a row group already satisfies the pushdown predicate (`fully_matched`, tracked by `ParquetAccessPlan` since apache#21637), running the per-row `RowFilter` inside that row group is pure overhead — every row passes anyway. This PR teaches the push-decoder stream to **toggle the `RowFilter` off across fully-matched runs** and back on at the next straddling row group, so fully-matched runs decode without per-row evaluation (and two-phase reads skip fetching filter-only columns for them). Common shape: time-partitioned tables (`WHERE ts >= X AND ts < Y` with whole row groups inside `[X, Y)`) and the middle stretch of a TopK scan. This PR was decomposed per [@adriangb's proposal](apache#23696 (comment)); the standalone pieces already merged: - apache#24509 — generic `strip_empty_row_groups` in `PreparedAccessPlan::prepare` (apache#24287) - apache#24572 — `InitialDecoderState` extraction (apache#24286) What remains here is the feature itself plus the prebuild split it consumes (the proposal's PR 2 is **folded in** rather than standalone: on `main`, `into_builder()` preserves the `RowFilter` across runtime-prune rebuilds, so a prebuilt candidate list would have no consumer and no perf story on its own — the per-RG toggle here is its only consumer). ## What changes are included in this PR? - `access_plan.rs` — `PreparedAccessPlan` carries per-RG `fully_matched` flags, mapped from the access plan **after** the (generic) empty-row-group strip; `reorder_by_statistics` / `reverse` permute them alongside the indexes. - `row_filter.rs` — split into `prebuild_row_filter_candidates` (once per file: conjunct split, candidate construction, `reassign_expr_columns`) and `row_filter_from_prebuilt` (per boundary: metric wiring + optional `required_bytes` ordering, no tree walks). The public `build_row_filter` is **reimplemented on top of the two**, so the open-time path and the per-RG rebuild path share one conjunct-split/order/metric implementation and cannot drift. `DatafusionArrowPredicate::try_new` becomes a `#[cfg(test)]` convenience. - `push_decoder.rs` — `RgPlanEntry.fully_matched`; `RowFilterContext` (the prebuilt list + settings) lets `rebuild_decoder_at_boundary` swap between the real filter and an empty one as the stream crosses row-group boundaries; at most one `into_builder` rebuild per boundary, shared with the runtime pruner. - `opener/mod.rs` — prebuilds once per file, installs the initial filter based on the first row group's `fully_matched` state (two new fields on `InitialDecoderState`). - `metrics.rs` — new `row_filter_skipped_fully_matched` counter, **registered lazily on first suppression** (`RowFilterSkippedFullyMatchedMetric`), following the `page_index_pages_skipped_by_fully_matched` pattern from apache#21637: no public field on `ParquetFileMetrics` (no API/semver change) and no zero-valued entries in `EXPLAIN ANALYZE` baselines — the previously-regenerated 41 baselines across 3 slt files are untouched by this PR. It counts *suppression events*, not row groups: a run of consecutive fully-matched RGs shares one toggle. Considered but deferred: carrying one `Vec<{index, fully_matched}>` instead of the two positionally-aligned vectors. Alignment is now maintained in exactly two adjacent permutations (`reorder_by_statistics`, `reverse`) plus a `debug_assert`, and the type change would churn ~40 assertion lines in `main`-side tests — the opposite of shrinking this PR. Happy to do it as a tiny follow-up. ## Are these changes tested? - `fully_matched_rgs_skip_row_filter` — 4 RGs, predicate `v >= 3 AND v <= 10` makes RG 0 and RG 3 straddlers with a fully-matched run in between, covering the full toggle lifecycle (ON → OFF → back ON); asserts correctness, that 11/12 are filtered out by the reinstalled filter, and `row_filter_skipped_fully_matched >= 1`. - `dynamic_row_group_pruning.slt` — an `EXPLAIN ANALYZE` case surfacing `row_filter_skipped_fully_matched=1`, so the optimization is observable from SQL and a regression shows up as the metric disappearing. - All existing `dynamic_row_group_pruning` integration tests (10), the `datafusion-datasource-parquet` lib suite (240), and the affected slt files pass; `clippy --all-targets --all-features -D warnings` clean. ## Are there any user-facing changes? - `row_filter_skipped_fully_matched` appears in `EXPLAIN ANALYZE` on parquet scans **only when the toggle actually fired**. - No public API changes; `build_row_filter` keeps its signature and behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
When DataFusion evaluates a Parquet scan with filter pushdown, it uses row group statistics to determine which row groups contain matching rows. The
RowGroupAccessPlanFilteralready tracks which row groups are "fully matched" — where statistics prove that all rows satisfy the predicate (viais_fully_matched).However, this information was not propagated downstream. Even for fully matched row groups:
This is especially costly when filter columns are expensive to decode (e.g., large strings) or when predicates are complex. Common real-world examples include time-range filters where entire row groups fall within the range, or
WHERE status != 'DELETED'on data with no deleted rows.What changes are included in this PR?
DataFusion changes (this PR)
row_group_filter.rs:RowGroupAccessPlanFilter::build()now returns(ParquetAccessPlan, Vec<usize>)— the access plan plus the indices of fully matched row groups.page_filter.rs:prune_plan_with_page_index()accepts afully_matched_row_groupsparameter and skips page-level pruning for those row groups.opener.rs: Wires fully matched row groups through the pipeline — passes them to page pruning and to theParquetPushDecoderBuilderviawith_fully_matched_row_groups().Arrow-rs dependency (apache/arrow-rs#9694)
The new
ArrowReaderBuilder::with_fully_matched_row_groups()API in arrow-rs allows skippingRowFilterevaluation during Parquet decoding for specified row groups. This PR uses[patch.crates-io]pointing to the arrow-rs fork branch until that PR is merged and released.Benchmark
Includes a criterion benchmark (
parquet_fully_matched_filter) usingParquetPushDecoderdirectly — the same code path DataFusion's async opener uses. Dataset: 20 row groups × 50K rows, with a 1KB string payload column and predicatex < 200(all row groups fully matched).Are these changes tested?
datafusion-datasource-parquettests pass (16 failures are pre-existing, caused by missingparquet-testingsubmodule)Are there any user-facing changes?
No user-facing API changes. This is a transparent performance optimization — queries that previously worked will now be faster when row group statistics prove all rows match the predicate.
Note: This PR depends on apache/arrow-rs#9694. Theall logic is on df side now[patch.crates-io]inCargo.tomlwill be removed once that arrow-rs change is released.