Repository navigation
Conversation
1 task done
sezruby
force-pushed
the
replace-partitioned-by-rewrite
branch
from
November 13, 2026 19:54
0351c6e to
c61cdd9
Compare
sezruby
force-pushed
the
replace-partitioned-by-cluster-by
branch
from
November 18, 2026 17:07
70a746e to
685eec2
Compare
…partition columns ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY failed when a data file does not store the partition columns. Rewrite those files first, with an internal REORG (new internal mode MATERIALIZE_PARTITION_COLUMNS, used like REWRITE_TYPE_WIDENING), while the table is still partitioned. The REORG commits like OPTIMIZE: dataChange=false, deletion vectors applied, row tracking preserved, batched with optimize.batchSize. Then a new transaction verifies the footers again and commits the unchanged metadata-only conversion, which still only re-adds files. Files added without the partition columns in the meantime fail the command. The command is validated before any file is rewritten. If delta.writePartitionColumnsToParquet is false, the rewritten files would not store the partition columns either, so the command fails with DELTA_REPLACE_PARTITIONED_BY_WRITE_PARTITION_COLUMNS_DISABLED instead. TransactionalWrite.writesPartitionColumns holds the rule for both. The internal conf spark.databricks.delta.alterTable.replacePartitionedBy.rewriteNonMaterializedFiles (default true) turns the rewrite off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Eunjin Song <sezruby@gmail.com>
sezruby
force-pushed
the
replace-partitioned-by-rewrite
branch
from
November 18, 2026 18:01
c61cdd9 to
84bad05
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
delta-io#8227 drops the partitioning with a metadata-only commit, so it requires every data file to store the partition columns. Files written before delta-io#6525, written with
delta.writePartitionColumnsToParquet = false, or added byCONVERT TO DELTAusually don't, and the command fails on them.With this change the command rewrites those files first, in separate commits, and then drops the partitioning as before:
MATERIALIZE_PARTITION_COLUMNS, the same wayDROP FEATURE typeWideningusesREWRITE_TYPE_WIDENING, so it commits like OPTIMIZE:REORG,dataChange = false;optimize.batchSize;The conversion commit still only re-adds files, so Kernel's incremental CRC computation (delta-io#8044), which treats this operation's added files as re-adds, stays correct. The history matches the same conversion in Databricks:
REORG, thenREPLACE PARTITIONED BY WITH CLUSTER BY.If
delta.writePartitionColumnsToParquetis set tofalse(and neither IcebergCompat normaterializePartitionColumnsis enabled), the rewritten files would not store the partition columns either. The command then fails with the new errorDELTA_REPLACE_PARTITIONED_BY_WRITE_PARTITION_COLUMNS_DISABLEDbefore committing anything, and asks to set the property totrue. The rule for whether a write stores the partition columns moves fromTransactionalWrite.writeFilestoTransactionalWrite.writesPartitionColumns, so both use the same check.The new internal conf
spark.databricks.delta.alterTable.replacePartitionedBy.rewriteNonMaterializedFiles(defaulttrue) turns the rewrite off; the command then fails withDELTA_REPLACE_PARTITIONED_BY_PARTITION_COLUMNS_NOT_MATERIALIZEDas before.How was this patch tested?
ReplacePartitionedByWithClusterBySuite:REORGcommit removes exactly the files without the partition column and adds files that store it, all withdataChange = false. The following conversion commit has noRemoveFileand re-adds every file. Files that already stored the column keep their paths. The rewritten files get stats for the former partition column, data skipping on it works, and results (includingIS NULLand equality filters) are unchanged._metadata.row_id/row_commit_versionare unchanged.DELTA_REPLACE_PARTITIONED_BY_PARTITION_COLUMNS_NOT_MATERIALIZED, table unchanged.delta.writePartitionColumnsToParquet = false: fails with the new error, no commit.REORGcommit is made when every file stores the partition columns.With the reorg file filter selecting every file, the mixed-table test fails (files that stored the column are rewritten too); without the
writePartitionColumnsToParquetcheck, the error test fails.Also ran
DeltaReorgSuite,MaterializePartitionColumnsFeatureSuite,TypeWideningTableFeatureSuite,DeltaSqlParserSuite,DeltaErrorsSuite,DeltaThrowableSuiteand scalastyle.Does this PR introduce any user-facing changes?
Yes.
ALTER TABLE ... REPLACE PARTITIONED BYno longer fails on tables whose data files do not store the partition columns: it rewrites those files withREORGcommits first. On a table withdelta.writePartitionColumnsToParquet = falseit fails with a new error that asks to enable the property.🤖 Generated with Claude Code