Skip to content

[Spark] Read CDF files with the partitioning they were written with - #23

Draft
sezruby wants to merge 3 commits into
masterfrom
cdf-read-across-partition-changes
Draft

sezruby wants to merge 3 commits into
masterfrom
cdf-read-across-partition-changes

Conversation

@sezruby

@sezruby sezruby commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

After a table's partition columns change, a batch Change Data Feed read of versions written before the change returns NULL for the columns whose partitioning changed. Time travel to the same version returns the correct values. Two cases are affected:

  1. A partition column becomes a data column, and the files written before the change do not store it (files written before delta.writePartitionColumnsToParquet existed, or with it set to false).
  2. A data column becomes a partition column. Files written before the change store the value but have no entry for it in partitionValues.

delta-io#7848 turns these reads into a DELTA_CHANGE_DATA_FEED_INCOMPATIBLE_SCHEMA_CHANGE error. Released versions return the NULL values without an error. One case still returns NULL without an error on master: a read that starts at the commit that changes the partition columns. That commit removes the files written with the previous partition columns, and its delete rows are read with the new ones. The delta-io#7848 checks compare the read snapshot with the start version's metadata, which already has the new partition columns, so the read is not blocked.

Cause: CDCReader.changesToDF builds every file index from one read snapshot, so every file is read with that snapshot's partition columns. A file's partitionValues holds the partition columns in effect when it was written.

This PR reads each file of a batch CDF scan with the partition columns it was written with. Those follow from the version of the file's change data spec, so no file is inspected:

  • A file that a version adds, and its change data files, are written with that version's partition columns.
  • A file that a version removes was written with the previous version's partition columns: an operation that changes the partition columns removes or re-adds every file, because partitionValues holds a value for each partition column.

CdfPartitionSchemaTracker (from delta-io#7848) already sees the metadata of every version in the range. It now also records the versions whose metadata changes the physical partition columns, and returns them (CdfWrittenPartitioning) when some file was written with other partition columns than the read snapshot. The change data specs of the scans (change data files, added and removed files, and the deletion-vector scans) are then grouped by version, and each group is scanned with a snapshot descriptor that differs from the read snapshot only in partitionColumns. Every scan has the same output columns, so the scans are unioned as before. When every file was written with the read snapshot's partitioning, nothing is grouped and the read is unchanged.

When the start version itself has a metadata action and removes files, the files it removes were written with the version before it. Its metadata is loaded only then, with the same condition as the starting-commit schema check.

Driver cost: the tracker compares the partition columns of each metadata action in the range; it does not visit file actions in this mode. Grouping looks up the partitioning once per change data spec (one per version and scan type) and moves specs by reference. A read whose files were all written with the read snapshot's partitioning skips both. The only extra I/O is loading the metadata of the version before the start, when the start version has a metadata action and removes files.

With the values read correctly, the partition-only check from delta-io#7848 no longer blocks these reads. Its block decision is unchanged when the new conf is disabled; the start snapshot's file count it uses is now only computed when the decision needs it. Other schema compatibility checks, including column mapping, type and nullability changes, are unchanged. The new internal conf spark.databricks.delta.changeDataFeed.readFilesWithWrittenPartitioning.enabled (default true) turns the new behavior off and restores the delta-io#7848 block.

Streaming CDF reads are not changed. The streaming source fails on partition column changes unless allowUnsafeStreamingReadOnPartitionColumnChanges is set.

How was this patch tested?

New tests in CDCReaderSuite:

  • CDF of versions before a partition column was removed returns the original values, with and without delta.writePartitionColumnsToParquet and deletion vectors, for INSERT, UPDATE and DELETE history, for a range that spans the change, and with filters on the former and the remaining partition column.
  • CDF of versions before a column became a partition column returns its values, with and without deletion vectors, for a range that spans the change and with a filter on the new partition column.
  • CDF across several partition column changes, (a) to (a, b) to none to (b), with data written under each, returns the original values for the whole range, a range in between, and with a filter, with and without delta.writePartitionColumnsToParquet.
  • The files written with the same partition columns are scanned together: one scan when all files share a partitioning, one scan per partitioning across the change.
  • CDF that starts at a partition column change reads the files that the change removes with the previous partitioning, with and without delta.writePartitionColumnsToParquet. On master this read is not blocked, and without stored partition columns its delete rows have NULL for the former partition column.
  • With the new conf disabled, the read is blocked as in [Spark] Block CDF batch reads across incompatible partition changes delta-io/delta#7848.

The delta-io#7848 tests that assert a blocked read now set the new conf to false, and each also checks that the same read returns the original rows when the conf is enabled.

build/sbt "spark/testOnly org.apache.spark.sql.delta.cdc.* org.apache.spark.sql.delta.DeltaCDC*" (25 suites, including deletion vectors, column mapping and streaming): 921 passed, 0 failed (3 canceled and 390 ignored by the existing suite mixins). CDCReaderSuite on the last commit: 70 passed, 0 failed. With the lookup of the version before the start disabled, the new start-version test fails when the files do not store the partition columns.

Does this PR introduce any user-facing changes?

Yes. Batch CDF reads of versions written before a partition schema change return the correct values instead of failing with DELTA_CHANGE_DATA_FEED_INCOMPATIBLE_SCHEMA_CHANGE (since delta-io#7848) or returning NULL (in released versions).

🤖 Generated with Claude Code

sezruby and others added 2 commits November 16, 2026 10:02
Batch CDF reads every file with the read snapshot's partition columns.
After a partition schema change, files written before the change have
other partitionValues keys, so a former partition column that the file
does not store, or a new partition column that the file stores as a data
column, is read as NULL. delta-io#7848 blocks these reads instead.

Group the files of each batch CDF scan by the keys of their
partitionValues and scan each group with those partition columns. Reads
that do not cross a partition schema change use a single group and are
unchanged. With the values read correctly, the partition-only check from
delta-io#7848 no longer blocks the read, and its tracker is not built. Streaming
reads are unchanged. The new internal conf
spark.databricks.delta.changeDataFeed.readFilesWithWrittenPartitioning.enabled
(default true) turns this off.

Signed-off-by: Eunjin Song <sezruby@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instead of inspecting the partitionValues keys of every file, take the
partition columns that a file was written with from the version of its
change data spec:

- A file that a version adds, and its change data files, are written with
  that version's partition columns.
- A file that a version removes was written with the previous version's
  partition columns: an operation that changes the partition columns
  removes or re-adds every file.

CdfPartitionSchemaTracker, which already sees the metadata of every
version for the delta-io#7848 check, now also records the versions whose metadata
changes the physical partition columns. When every file was written with
the read snapshot's partitioning, the specs are not grouped and the read is
unchanged; otherwise the specs are grouped by version. No file is
inspected, so the extra driver work no longer grows with the number of
files in the range.

When the start version has a metadata action and removes files, its
removed files are read with the metadata of the version before it, loaded
only in that case. The tracker's start snapshot file count is now only
computed when its block decision needs it.

Signed-off-by: Eunjin Song <sezruby@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sezruby
sezruby force-pushed the cdf-read-across-partition-changes branch from 401450d to 7479e48 Compare November 16, 2026 18:22
… written partitioning

CdfPartitionSchemaTracker.processActions only feeds the block decision of
the delta-io#7848 check, which does not apply when files are read with the
partitioning they were written with. Skip it in that mode, so that this
mode only looks at the metadata of the versions in the range and adds no
work per file.

Signed-off-by: Eunjin Song <sezruby@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant