fix: prune row groups when file statistics collapse the predicate to a constant

[jensholdgaard]

Note

This fix was investigated and written with AI assistance (Claude Code), posted with the account owner's review and consent.

Which issue does this PR close?

• Closes Regression: DataFusion 55 no longer prunes row groups for col = <literal> when statistics show the column is entirely NULL #24769.

Rationale for this change

A query like SELECT ... WHERE col = 'x' over a Parquet file whose statistics say col is entirely NULL cannot match any row, and DataFusion 54 pruned such row groups from the footer without reading them. On 55 the same query reads them.

The cause is an interaction rather than a bug in pruning itself. constant_columns_from_stats substitutes columns that file statistics prove constant, and its all-NULL branch folds such a column to a NULL literal. Once substituted, the predicate simplifies to a bare constant — NULL here — at which point build_pruning_predicates returns None (there are no column references left to build a pruning predicate over), and prune_row_groups falls through with no pruning at all.

So for exactly the files where the statistics carry the most information, the substitution is strictly counterproductive: before it, the pruning predicate's own col_null_count != row_count conjunct proved the row group empty and skipped it.

This surfaced as a regression when bisecting a real workload (a Parquet log store where a body column is NULL for the vast majority of rows) from 54 → 55. The bisect lands on #22969, which removed ListingOptions::collect_stat in favour of the session's execution.collect_statistics — default true. That change is correct in itself; it simply began feeding per-file statistics to the substitution on paths that previously had none, exposing the gap. Setting execution.collect_statistics = false restores pruning on 55, which is a useful confirmation but obviously not a fix.

Results were never wrong — a filter drops NULL and false rows alike — but the scan work is real: row groups that used to be skipped from the footer are now decoded in full.

What changes are included in this PR?

prune_row_groups now recognises the collapsed-to-constant case: if the (post-substitution, post-simplification) predicate is a false or NULL literal, every remaining row group is skipped and the skip is credited to row_groups_pruned_statistics. A small RowGroupAccessPlanFilter::skip_all helper is added alongside the existing prune_by_* methods.

The check is deliberately narrow — a downcast to Literal plus a NULL/false value test — so it cannot affect predicates that still reference columns; those take the existing path unchanged.

Are these changes tested?

Yes: test_prune_all_null_column_equality_from_file_statistics in opener/mod.rs, modelled on the neighbouring test_prune_on_partition_values_and_file_statistics. It fails on current main (3 rows scanned, 0 pruned) and passes with this change.

One note on how it asserts, in case it saves a reviewer time: it checks the row_groups_pruned_statistics metric rather than the returned row count. A row-count assertion cannot distinguish "pruned" from "scanned, then row-filtered" — both give zero rows — and an earlier draft of this test passed against the unfixed code for exactly that reason.

Verified cargo test -p datafusion-datasource-parquet --lib opener:: — 47 passed. The 16 pre-existing bloom_filter::tests failures on this crate are unrelated and reproduce identically on an untouched checkout.

Are there any user-facing changes?

No API or result changes. Queries that were already correct stay correct; affected scans read less data. Plans may show more row groups pruned, and row_groups_pruned_statistics increases correspondingly.

[jensholdgaard]

The cargo test (amd64) failure was real and pointed at a scoping problem in my first push — fixed in the update.

The nine failing evolved_schema* / test_pushdown_with_missing_* tests exercise the other path that substitutes NULL for a column: files missing the column entirely (schema evolution). My original check keyed only on "the simplified predicate is a constant NULL/false literal", so it also fired there, pruning row groups those tests expect to see in the raw scan output and shifting their pushdown metric counts. Semantically that pruning would be sound too (a file missing c2 cannot satisfy c2 = 2), but it is a behaviour change well beyond the regression this PR fixes, and partition-value-driven collapse is similarly already FilePruner's territory.

The update narrows the skip with a stats_constants_in_predicate guard: the collapse only counts as statistics-proven when the original predicate referenced a column that constant_columns_from_stats proved constant for this file. With that:

• the nine previously failing tests pass again unchanged,
• the new regression test still fails without the fix and passes with it,
• cargo test -p datafusion-datasource-parquet --lib opener:: — 47 passed,
• cargo test -p datafusion --lib datasource::physical_plan::parquet::tests — 35 passed (the 4 parquet_exec_with_* failures on my machine reproduce on an untouched checkout and pass in CI, so they are environmental).

If maintainers would rather generalise the skip to the missing-column collapse as well, I'm happy to do that in a follow-up with the corresponding test updates — it just seemed wrong to smuggle a second behaviour change into a targeted regression fix.

[kosiew]

@jensholdgaard,

Thanks for working on this. The overall direction makes sense, and restoring pruning for predicates that collapse to false or NULL from file statistics addresses the regression nicely.

I found one case that I think needs to be addressed before merging. The current guard tracks whether the predicate references any stats-derived constant, but not whether that substitution is actually what made the predicate unsatisfiable. This can change behavior for mixed predicates involving schema-evolution missing columns.

I also left a small non-blocking test suggestion to cover the non-NULL false path.

[kosiew]

I think this guard is a little too broad. stats_constants_in_predicate tells us that the predicate references some column proven constant by file statistics, but it does not tell us that the stats substitution is what caused the final predicate to become false or NULL.

For example, consider a = 1 AND missing_b = 2 on a file where statistics prove a = 1. This guard passes because a is a stats-derived constant. Stats folding leaves that conjunct as true, but the missing-column adapter can then fold missing_b = 2 to NULL or false. We would then skip all row groups even though the stats substitution itself did not prove the predicate unsatisfiable.

Before this change, that case follows the intentionally preserved missing-column path and scans/filter-prunes instead. Could we track whether the stats substitution itself makes the predicate unsatisfiable, before missing-column or partition rewriting, or otherwise exclude this mixed case?

It would also be good to add a regression test with one stats-constant present column and one missing column.

[jensholdgaard]

Written by Claude, an AI assistant, on behalf of the PR author.

Agreed, and addressed in 6f86ec7. The flag recorded only that the predicate referenced a column statistics had proven constant, then treated any later collapse to false/NULL as proof that statistics caused it. Those are two different claims, and the a = 1 AND missing_b = 2 example separates them.

The check now asks the narrower question at substitution time: substitute only the statistics-derived constants, simplify that, and take a collapse as proof. Anything the missing-column adapter or partition folding does afterwards is irrelevant, because the proof already stands on the file's own statistics. In the mixed case, missing_b = 2 is left standing as a real column reference, nothing collapses, and the file takes the existing missing-column path. That also drops the Literal downcast from prune_row_groups, which is now just the two flags; collect_columns stays as the cheap short-circuit so the simplifier only runs when a stats constant is actually referenced. The simplifier is given the full table schema, since the substitution deliberately leaves non-constant columns in place and it types each node as it walks.

test_no_prune_when_missing_column_collapses_mixed_predicate covers exactly that shape: a stats-constant present column plus a missing column, asserting the statistics branch does not claim the prune and the file is scanned (3 rows returned, 0 row groups pruned). The evolved_schema* and test_pushdown_with_missing_* tests are unchanged and pass.

[kosiew]

Nice to have coverage for the all-NULL case. Could we also add a small test where exact non-NULL min/max statistics prove a column constant and a = <different literal> simplifies to false? That would explicitly cover both the NULL and false branches handled by the new pruning logic.

[jensholdgaard]

Written by Claude, an AI assistant, on behalf of the PR author.

Added in 6f86ec7 as test_prune_exact_constant_column_false_predicate_from_file_statistics: exact min == max == 7 statistics prove a constant, and a = 8 folds to false, so the row group is pruned on that proof. Together with the all-NULL case, both literal values the branch accepts are now covered. The row_groups_pruned_statistics accessor is hoisted into a shared test helper, since all three tests assert on that metric rather than the row count, which cannot tell "pruned" from "scanned, then row-filtered".

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.93048% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.59%. Comparing base (9ef3d47) to head (6f86ec7).
⚠️ Report is 71 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/datasource-parquet/src/opener/mod.rs	98.89%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24770      +/-   ##
==========================================
+ Coverage   81.52%   81.59%   +0.06%     
==========================================
  Files        1123     1123              
  Lines      405970   411094    +5124     
  Branches   405970   411094    +5124     
==========================================
+ Hits       330983   335437    +4454     
- Misses      55626    55910     +284     
- Partials    19361    19747     +386     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kosiew]

@jensholdgaard,

Thanks for the follow-up here. The new stats_prove_unsatisfiable check is heading in the right direction, and I like that it is computed before the partition and schema adapters run. The added NULL and exact non-NULL cases also cover the two constant outcomes well.

I think there is still one blocking schema-evolution case, though. Collected file statistics can represent a column that is physically absent from the Parquet file as all NULL, which means that missing column can still enter stats_constants. In that case the new statistics-only check can classify the predicate as unsatisfiable and skip the row group, even though the intent is to preserve the existing missing-column behavior.

The mixed regression test does not currently exercise that path because the missing column's statistics are left unknown. I think the fix needs to distinguish an all-NULL constant that came from an actually present physical column from one synthesized for an absent column, or otherwise defer this attribution until the physical schema is known.

Once that case is covered, this should be in much better shape.

[kosiew]

I think there is still a schema-evolution hole here. DFParquetMetadata::statistics_from_parquet_metadata gives a logical column that is absent from the physical Parquet schema null_count = Exact(num_rows). constant_value_from_stats then turns that into a NULL entry in stats_constants.

So for a file that physically contains only a, collected stats can still make missing b look like a stats-derived NULL constant. With a = 1 AND b = 2, stats_alone_unsatisfiable can simplify the whole predicate to NULL, set stats_prove_unsatisfiable, and skip the row group through the statistics path.

That seems to bring back the missing-column row-group skip that this change is trying to avoid. The new mixed regression currently leaves b statistics unknown, so it does not reproduce what real collected schema-evolution statistics provide.

Could we preserve whether an all-NULL constant came from a physically absent column, or defer this attribution until the physical schema is known? I would also update the regression so b.null_count = Exact(3), or better, use real collected statistics for the missing-column case.