[Parquet] Add row group distinct counts to StatisticsConverter - #10652
[Parquet] Add row group distinct counts to StatisticsConverter#10652Rich-T-kid wants to merge 4 commits into
StatisticsConverter#10652Conversation
Adds `StatisticsConverter::row_group_distinct_counts()` which reads the `distinct_count` field from parquet row group statistics and returns a `UInt64Array` with one entry per row group (null where the field is absent). This mirrors the existing `row_group_null_counts` and `row_group_nan_counts` methods and unblocks DataFusion from populating `ColumnStatistics::distinct_count` via `statistics_from_parquet_metadata`. Previously this was always `Precision::Absent` for parquet files regardless of whether the file contained the field. Non-arrow-rs writers (parquet-java, parquet-cpp) may already populate `distinct_count` in the file footer, so the read path is immediately useful without any write-side changes.
|
🤔 datafusion does do something similar here https://github.com/apache/datafusion/blob/041a71671c7a505a2e3ed58689a20137c8f3d172/datafusion/datasource-parquet/src/metadata.rs#L835, but with this PR we allow arrow-rs user to do it directly as well make a consistent api. |
etseidl
changed the title
StatisticsConverter
etseidl
left a comment
etseidl
left a comment
There was a problem hiding this comment.
This seems sensible to me, esp. if there are writers that actually populate this statistic.
How to tackle the write side of this is still a matter of debate. 😅
@etseidl i just ran a simple implementation gated by a boolean flag and it ranges from 1.4x-8.8x slower depending on the data type. |
That's quite the hit, glad it's gated. But I can see a use for write-once-read-many scenarios where you'd likely want the cardinality and are willing to pay for it. |
|
I'll try and see if I can make it more performant and push up the PR tomorrow 👍 |
2 participants
Which issue does this PR close?
Rationale for this change
see #10650
What changes are included in this PR?
Added StatisticsConverter::row_group_distinct_counts() to the parquet crate, which reads the distinct_count field from row group statistics and returns a UInt64Array with one entry per row group.
Are these changes tested?
yes this PR includes two test that assert that row group stats are being propogated
Are there any user-facing changes?
yes, adds new public method for reading row group distinct stats