Repository navigation
Conversation
Batch CDF reads every file with the read snapshot's partition columns. After a partition schema change, files written before the change have other partitionValues keys, so a former partition column that the file does not store, or a new partition column that the file stores as a data column, is read as NULL. delta-io#7848 blocks these reads instead. Group the files of each batch CDF scan by the keys of their partitionValues and scan each group with those partition columns. Reads that do not cross a partition schema change use a single group and are unchanged. With the values read correctly, the partition-only check from delta-io#7848 no longer blocks the read, and its tracker is not built. Streaming reads are unchanged. The new internal conf spark.databricks.delta.changeDataFeed.readFilesWithWrittenPartitioning.enabled (default true) turns this off. Signed-off-by: Eunjin Song <sezruby@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instead of inspecting the partitionValues keys of every file, take the partition columns that a file was written with from the version of its change data spec: - A file that a version adds, and its change data files, are written with that version's partition columns. - A file that a version removes was written with the previous version's partition columns: an operation that changes the partition columns removes or re-adds every file. CdfPartitionSchemaTracker, which already sees the metadata of every version for the delta-io#7848 check, now also records the versions whose metadata changes the physical partition columns. When every file was written with the read snapshot's partitioning, the specs are not grouped and the read is unchanged; otherwise the specs are grouped by version. No file is inspected, so the extra driver work no longer grows with the number of files in the range. When the start version has a metadata action and removes files, its removed files are read with the metadata of the version before it, loaded only in that case. The tracker's start snapshot file count is now only computed when its block decision needs it. Signed-off-by: Eunjin Song <sezruby@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sezruby
force-pushed
the
cdf-read-across-partition-changes
branch
from
November 16, 2026 18:22
401450d to
7479e48
Compare
… written partitioning CdfPartitionSchemaTracker.processActions only feeds the block decision of the delta-io#7848 check, which does not apply when files are read with the partitioning they were written with. Skip it in that mode, so that this mode only looks at the metadata of the versions in the range and adds no work per file. Signed-off-by: Eunjin Song <sezruby@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
After a table's partition columns change, a batch Change Data Feed read of versions written before the change returns
NULLfor the columns whose partitioning changed. Time travel to the same version returns the correct values. Two cases are affected:delta.writePartitionColumnsToParquetexisted, or with it set tofalse).partitionValues.delta-io#7848 turns these reads into a
DELTA_CHANGE_DATA_FEED_INCOMPATIBLE_SCHEMA_CHANGEerror. Released versions return theNULLvalues without an error. One case still returnsNULLwithout an error on master: a read that starts at the commit that changes the partition columns. That commit removes the files written with the previous partition columns, and its delete rows are read with the new ones. The delta-io#7848 checks compare the read snapshot with the start version's metadata, which already has the new partition columns, so the read is not blocked.Cause:
CDCReader.changesToDFbuilds every file index from one read snapshot, so every file is read with that snapshot's partition columns. A file'spartitionValuesholds the partition columns in effect when it was written.This PR reads each file of a batch CDF scan with the partition columns it was written with. Those follow from the version of the file's change data spec, so no file is inspected:
partitionValuesholds a value for each partition column.CdfPartitionSchemaTracker(from delta-io#7848) already sees the metadata of every version in the range. It now also records the versions whose metadata changes the physical partition columns, and returns them (CdfWrittenPartitioning) when some file was written with other partition columns than the read snapshot. The change data specs of the scans (change data files, added and removed files, and the deletion-vector scans) are then grouped by version, and each group is scanned with a snapshot descriptor that differs from the read snapshot only inpartitionColumns. Every scan has the same output columns, so the scans are unioned as before. When every file was written with the read snapshot's partitioning, nothing is grouped and the read is unchanged.When the start version itself has a metadata action and removes files, the files it removes were written with the version before it. Its metadata is loaded only then, with the same condition as the starting-commit schema check.
Driver cost: the tracker compares the partition columns of each metadata action in the range; it does not visit file actions in this mode. Grouping looks up the partitioning once per change data spec (one per version and scan type) and moves specs by reference. A read whose files were all written with the read snapshot's partitioning skips both. The only extra I/O is loading the metadata of the version before the start, when the start version has a metadata action and removes files.
With the values read correctly, the partition-only check from delta-io#7848 no longer blocks these reads. Its block decision is unchanged when the new conf is disabled; the start snapshot's file count it uses is now only computed when the decision needs it. Other schema compatibility checks, including column mapping, type and nullability changes, are unchanged. The new internal conf
spark.databricks.delta.changeDataFeed.readFilesWithWrittenPartitioning.enabled(defaulttrue) turns the new behavior off and restores the delta-io#7848 block.Streaming CDF reads are not changed. The streaming source fails on partition column changes unless
allowUnsafeStreamingReadOnPartitionColumnChangesis set.How was this patch tested?
New tests in
CDCReaderSuite:delta.writePartitionColumnsToParquetand deletion vectors, for INSERT, UPDATE and DELETE history, for a range that spans the change, and with filters on the former and the remaining partition column.delta.writePartitionColumnsToParquet.delta.writePartitionColumnsToParquet. On master this read is not blocked, and without stored partition columns its delete rows haveNULLfor the former partition column.The delta-io#7848 tests that assert a blocked read now set the new conf to
false, and each also checks that the same read returns the original rows when the conf is enabled.build/sbt "spark/testOnly org.apache.spark.sql.delta.cdc.* org.apache.spark.sql.delta.DeltaCDC*"(25 suites, including deletion vectors, column mapping and streaming): 921 passed, 0 failed (3 canceled and 390 ignored by the existing suite mixins).CDCReaderSuiteon the last commit: 70 passed, 0 failed. With the lookup of the version before the start disabled, the new start-version test fails when the files do not store the partition columns.Does this PR introduce any user-facing changes?
Yes. Batch CDF reads of versions written before a partition schema change return the correct values instead of failing with
DELTA_CHANGE_DATA_FEED_INCOMPATIBLE_SCHEMA_CHANGE(since delta-io#7848) or returningNULL(in released versions).🤖 Generated with Claude Code