Skip to content

[Spark] REPLACE PARTITIONED BY: rewrite data files that do not store partition columns - #18

Draft
sezruby wants to merge 1 commit into
replace-partitioned-by-cluster-byfrom
replace-partitioned-by-rewrite
Draft

sezruby wants to merge 1 commit into
replace-partitioned-by-cluster-byfrom
replace-partitioned-by-rewrite

Conversation

@sezruby

@sezruby sezruby commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Stacked on delta-io#8227 (ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY), the follow-up described in delta-io#8226. Only the last commit is new.

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

delta-io#8227 drops the partitioning with a metadata-only commit, so it requires every data file to store the partition columns. Files written before delta-io#6525, written with delta.writePartitionColumnsToParquet = false, or added by CONVERT TO DELTA usually don't, and the command fails on them.

With this change the command rewrites those files first, in separate commits, and then drops the partitioning as before:

  1. The command is validated as before (partitioned table, no UniForm or adaptive metadata, clustering columns), so an invalid command fails before any file is rewritten.
  2. The Parquet footers are read. If every file stores the partition columns, the metadata-only conversion runs unchanged.
  3. Otherwise an internal REORG rewrites only the files that do not store all partition columns, while the table is still partitioned. It uses a new internal reorg mode, MATERIALIZE_PARTITION_COLUMNS, the same way DROP FEATURE typeWidening uses REWRITE_TYPE_WIDENING, so it commits like OPTIMIZE:
    • operation REORG, dataChange = false;
    • deletion vectors are applied, and row IDs and row commit versions are preserved;
    • the commits are batched with optimize.batchSize;
    • concurrent appends do not conflict with it.
  4. A new transaction then verifies the footers again and commits the metadata-only conversion. If files without the partition columns were added in the meantime, the command fails instead of rewriting again.

The conversion commit still only re-adds files, so Kernel's incremental CRC computation (delta-io#8044), which treats this operation's added files as re-adds, stays correct. The history matches the same conversion in Databricks: REORG, then REPLACE PARTITIONED BY WITH CLUSTER BY.

If delta.writePartitionColumnsToParquet is set to false (and neither IcebergCompat nor materializePartitionColumns is enabled), the rewritten files would not store the partition columns either. The command then fails with the new error DELTA_REPLACE_PARTITIONED_BY_WRITE_PARTITION_COLUMNS_DISABLED before committing anything, and asks to set the property to true. The rule for whether a write stores the partition columns moves from TransactionalWrite.writeFiles to TransactionalWrite.writesPartitionColumns, so both use the same check.

The new internal conf spark.databricks.delta.alterTable.replacePartitionedBy.rewriteNonMaterializedFiles (default true) turns the rewrite off; the command then fails with DELTA_REPLACE_PARTITIONED_BY_PARTITION_COLUMNS_NOT_MATERIALIZED as before.

How was this patch tested?

ReplacePartitionedByWithClusterBySuite:

  • Mixed table (two files without and two with the partition column, including a null partition): one REORG commit removes exactly the files without the partition column and adds files that store it, all with dataChange = false. The following conversion commit has no RemoveFile and re-adds every file. Files that already stored the column keep their paths. The rewritten files get stats for the former partition column, data skipping on it works, and results (including IS NULL and equality filters) are unchanged.
  • Deletion vectors + row tracking: rewritten files have no DV, deleted rows stay deleted, and _metadata.row_id / row_commit_version are unchanged.
  • Rewrite disabled: fails with DELTA_REPLACE_PARTITIONED_BY_PARTITION_COLUMNS_NOT_MATERIALIZED, table unchanged.
  • delta.writePartitionColumnsToParquet = false: fails with the new error, no commit.
  • Invalid command (non-existing clustering column) on a table that needs the rewrite: fails before any file is rewritten.
  • The existing metadata-only test checks that no REORG commit is made when every file stores the partition columns.

With the reorg file filter selecting every file, the mixed-table test fails (files that stored the column are rewritten too); without the writePartitionColumnsToParquet check, the error test fails.

Also ran DeltaReorgSuite, MaterializePartitionColumnsFeatureSuite, TypeWideningTableFeatureSuite, DeltaSqlParserSuite, DeltaErrorsSuite, DeltaThrowableSuite and scalastyle.

Does this PR introduce any user-facing changes?

Yes. ALTER TABLE ... REPLACE PARTITIONED BY no longer fails on tables whose data files do not store the partition columns: it rewrites those files with REORG commits first. On a table with delta.writePartitionColumnsToParquet = false it fails with a new error that asks to enable the property.

🤖 Generated with Claude Code

@sezruby
sezruby force-pushed the replace-partitioned-by-rewrite branch from 0351c6e to c61cdd9 Compare November 13, 2026 19:54
@sezruby
sezruby force-pushed the replace-partitioned-by-cluster-by branch from 70a746e to 685eec2 Compare November 18, 2026 17:07
…partition columns

ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY failed when a data
file does not store the partition columns. Rewrite those files first,
with an internal REORG (new internal mode MATERIALIZE_PARTITION_COLUMNS,
used like REWRITE_TYPE_WIDENING), while the table is still partitioned.
The REORG commits like OPTIMIZE: dataChange=false, deletion vectors
applied, row tracking preserved, batched with optimize.batchSize. Then a
new transaction verifies the footers again and commits the unchanged
metadata-only conversion, which still only re-adds files. Files added
without the partition columns in the meantime fail the command.

The command is validated before any file is rewritten. If
delta.writePartitionColumnsToParquet is false, the rewritten files would
not store the partition columns either, so the command fails with
DELTA_REPLACE_PARTITIONED_BY_WRITE_PARTITION_COLUMNS_DISABLED instead.
TransactionalWrite.writesPartitionColumns holds the rule for both.

The internal conf
spark.databricks.delta.alterTable.replacePartitionedBy.rewriteNonMaterializedFiles
(default true) turns the rewrite off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Eunjin Song <sezruby@gmail.com>
@sezruby
sezruby force-pushed the replace-partitioned-by-rewrite branch from c61cdd9 to 84bad05 Compare November 18, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant