Skip to content

[Spark] Support ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY - #17

Draft
sezruby wants to merge 44 commits into
masterfrom
replace-partitioned-by-cluster-by
Draft

sezruby wants to merge 44 commits into
masterfrom
replace-partitioned-by-cluster-by

Conversation

@sezruby

@sezruby sezruby commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Which Delta project/connector is this regarding?

  • Spark

Description

Adds a metadata-only way to remove the partitioning of a Delta table, optionally replacing it with liquid clustering:

ALTER TABLE t REPLACE PARTITIONED BY WITH CLUSTER BY NONE        -- plain unpartitioned table
ALTER TABLE t REPLACE PARTITIONED BY WITH CLUSTER BY (c1, c2)    -- clustered table

Today the only way to drop partitioning is to rewrite the whole table (CTAS / REPLACE TABLE). For large tables that are over-partitioned (e.g. partitioned by a high-cardinality ingestion id), that is very expensive. However, since delta.writePartitionColumnsToParquet (delta-io#6525, default on) / materializePartitionColumns (delta-io#5636), data files already contain the partition column values physically, so the partitioning can be dropped by only rewriting the log.

The syntax pairs REPLACE PARTITIONED BY with the existing CLUSTER BY (...) | CLUSTER BY NONE clause rather than introducing a new standalone command, so it composes with the existing clustering DDL.

Semantics

  1. Fails if the table is not partitioned.
  2. Reads the Parquet footer (schema only) of every data file and verifies that it contains all (physical) partition columns. If any file does not, fails with DELTA_REPLACE_PARTITIONED_BY_PARTITION_COLUMNS_NOT_MATERIALIZED, reporting the count and example paths. (Internal conf spark.databricks.delta.alterTable.replacePartitionedBy.verifyMaterializedPartitionColumns can disable this check.)
  3. Commits a single metadata-only version (via commitLarge, like RESTORE/CLONE, so any concurrent commit aborts it):
    • metaData with partitionColumns = [] (+ clustering table properties when columns are given);
    • every file re-added with partitionValues = {}, dataChange = false, and stats extended with minValues/maxValues/nullCount for the former partition columns (string/integral/decimal/date; null count only for other types), so data skipping on former partition columns continues to work. Partition values are parsed the same way they are read as partition values. An empty string (which the protocol defines as null, but which Delta reads as "" for string columns) or a value that cannot be cast to the column type gets no stats, since the value stored in the file is not known;
    • if a former partition column would otherwise lose stats (it is after the first delta.dataSkippingNumIndexedCols columns, or delta.dataSkippingStatsColumns is set), delta.dataSkippingStatsColumns is set to the columns that had stats before followed by the former partition columns. Partition columns cannot be listed in that property while the table is partitioned. Without this, a wide table would silently lose data skipping on the former partition columns, and CLUSTER BY (<former partition column>) would fail the stats-schema check;
    • clustering domain metadata when columns are given.
  4. CLUSTER BY NONE does not change the protocol. CLUSTER BY (cols) enables the clustering table feature, same as ALTER TABLE ... CLUSTER BY.
  5. Deletion vectors, row tracking (baseRowId, defaultRowCommitVersion) and column mapping are preserved. Time travel to pre-conversion versions keeps working.

Follow-ups / limitations

  • Tables with files that do not store partition columns need a rewrite first (e.g. a REORG that materializes them), which could be added later.
  • Timestamp/float/boolean/binary former partition columns get nullCount stats only.
  • Like any change of the partitioning, the conversion stops streaming readers of the table: a plain stream fails with DELTA_SCHEMA_CHANGED_WITH_VERSION on every restart and must be restarted with a new checkpoint. A stream with schemaTrackingLocation fails once with DELTA_STREAMING_METADATA_EVOLUTION and continues after a restart. Relaxing the check is not safe: a stream catching up from an older version also reads files that were removed before the conversion, and those were never checked for the partition columns.

How was this patch tested?

New ReplacePartitionedByWithClusterBySuite (parser, no-rewrite NONE conversion, stats + data skipping, writes/UPDATE/DELETE/MERGE/OPTIMIZE after conversion, conversion to clustered, catalog tables with null partitions, column mapping, DV + row tracking, error cases, empty-string partition values, wide tables (delta.dataSkippingStatsColumns set; stats kept for a partition column before and after the 32nd column, for old and new files), an existing delta.dataSkippingStatsColumns, streaming queries with and without schema tracking). Also ran DeltaSqlParserSuite, DeltaErrorsSuite, DeltaThrowableSuite and scalastyle.

Does this PR introduce any user-facing changes?

Yes, new SQL syntax ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY (...) | NONE.


Follow-up (stacked): #18 rewrites data files that do not store the partition columns instead of failing.

tdas and others added 29 commits November 17, 2026 02:02
…hema in the V2 connector (delta-io#8105)

[AI-assisted]

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

The V2 connector's `SchemaUtils` has a Kernel-to-Spark schema converter:
`convertKernelSchemaToSparkSchema`,
`convertKernelDataTypeToSparkDataType`,
`convertKernelFieldMetadataToSparkMetadata`, and their private helpers.
Main code never calls them. Table schemas are built with
`convertKernelMetadataToSparkSchema`, which parses the log's schema
string with `DataType.fromJson`. Since delta-io#7942, the converter also rejects
collated strings.

This PR:
- removes the unused converter and its helpers. The Spark-to-Kernel
direction is unchanged.
- removes the `SchemaUtilsTest` cases that only covered the removed
code, and the Kernel-to-Spark halves of the round-trip checks.
- changes `PartitionUtilsTest` to get a snapshot's Spark schema through
`convertKernelMetadataToSparkSchema`, as the connector does.

## How was this patch tested?

`build/sbt "sparkV2/Test/javafmtCheck" "sparkV2/javafmtCheck"
"sparkV2/testOnly org.apache.spark.sql.delta.v2.utils.SchemaUtilsTest
org.apache.spark.sql.delta.v2.utils.PartitionUtilsTest"`: formatting
checks are clean, and 65 JUnit cases pass (`SchemaUtilsTest` 42,
`PartitionUtilsTest` 23).

## Does this PR introduce _any_ user-facing changes?

No.

---------

Signed-off-by: tdas <tathagata.das1565@gmail.com>
…-io#8171)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Before writing data files, Delta makes the query's output attributes
nullable (`TransactionalWrite.makeOutputNullable`), so that a query that
under-reports nullability can't produce unreadable Parquet files. The
helper it uses, `SchemaUtils.typeAsNullable`, skips primitive map values
and the elements of nested arrays and maps. When such a field holds
NULLs that the query reports as NOT NULL (e.g. from `lag`/`lead` with a
NULL default), they land in a required Parquet field: the commit
succeeds, but reading the table fails. This affects CTAS, RTAS, INSERT
INTO, INSERT OVERWRITE, DataFrame writes and MERGE INSERT.

This PR adds `SchemaUtils.typeAsFullyNullable`, which makes every nested
field nullable, and uses it in `makeOutputNullable`. Only the Parquet
schemas of the data files change; the table schema in the Delta log was
already fully nullable.

`typeAsNullable` is unchanged, because the DDL NOT NULL validation
(`removeUnenforceableNotNullConstraints`) also uses it, and making it
fully recursive would reject table schemas that are accepted today. Its
doc now says what it leaves as it is.

## How was this patch tested?

New tests in `SchemaUtilsSuite` cover what `typeAsNullable` and
`typeAsFullyNullable` return for each kind of nested type.

New tests in `TransactionalWriteSuite` write a query whose `lag`/`lead`
defaults put NULLs into a primitive map value, nested array elements and
values nested in map values, which the query reports as NOT NULL:
- For each write (CTAS, RTAS, CREATE OR REPLACE TABLE AS SELECT, INSERT
INTO, INSERT OVERWRITE, DataFrame save/append/overwrite,
DataFrameWriterV2 append, MERGE INSERT, INSERT BY NAME, MERGE UPDATE,
UPDATE), the NULLs read back correctly.
- A check that the query reports these fields as NOT NULL but returns
NULLs there, so the tests keep exercising the bug.

## Does this PR introduce _any_ user-facing changes?

Yes, a bug fix: writes whose query reports a primitive map value or a
nested array/map element as NOT NULL, but returns NULLs there, used to
commit Parquet files that couldn't be read afterwards. They now produce
readable files, and the NULLs read back.
…implementation (delta-io#8149)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Reconstruct the change files of an AMT commit per state reconstruction
implementation.

The change file reconstruction decodes each manifest row into an
`AMTDataEntryExtended` and converts it with `DataEntryResolver`, while
the columnar state reconstruction builds the same `AddFile`s as columns.
This gives the columnar implementation a change file read built from
those columns, so it has a single `AddFile` derivation. The per-row
implementation keeps its read and stays the default and the fallback.

Review it commit by commit in the stacked history this was exported
from; the change is:

- `trackingDerivedChangeActions` and `mdvDerivedChangeActions` are
declared on `AMTCheckpointProviderImpl`. `PerRowAMTCheckpointProvider`
keeps the existing per-row code. `ColumnarAMTCheckpointProvider` builds
them from its `AddFile` columns.
- Columnar change files: a live entry becomes its `add` column, stamped
with the commit-level `dataChange` and without a back reference. Any
other entry becomes a `RemoveFile` built from that `add` the way
`removeWithTimestamp` builds it. `removeFileColumn` maps every
`RemoveFile` field explicitly, so a new field fails the read until it is
mapped.
- `rootDataEntries` and `leafDataEntries` are shared by the snapshot
reads, the MDV read and the tracking-derived change file read.
`leafEntriesWithRowIdPrefix` is replaced by `leafDataEntries`.
- `PerRowAMTCheckpointProvider` and `ColumnarAMTCheckpointProvider` move
into their own files; `AMTCheckpointProvider` is no longer sealed.

## How was this patch tested?

`AMTCommitReconstructionSuite` runs every test with both implementations
and compares the reconstructed change files with the committed ones
(adds, removes and MDV-derived removes). It also asserts the two
implementations produce identical actions field by field, and that each
reconstructed `RemoveFile` equals `AddFile.removeWithTimestamp` of the
previous `AddFile`. `AMTAddFileFieldCoverageSuite` pins the `RemoveFile`
fields and calls `removeFileColumn` for each.
`AMTAddFileReconstructionSuite` checks that both implementations reject
a leaf entry read from a manifest that is not one of its parent's
leaves.

## Does this PR introduce _any_ user-facing changes?

No.

Co-authored-by: Isaac <no-reply@databricks.com>
#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other

## Description

Convert `SnapshotManagerFactory` from a Java utility class to a Scala
object and accept `Option[CatalogTable]`. Scala callers can pass catalog
metadata directly, while Java callers use Scala `Option`. Preserve
catalog selection, lookup timing, exceptions, and lazy path-based
fallback.

Keep the factory scoped to the v2 connector. Adapt the existing cache
tests to scoped Scala-object spying, add the required test dependency,
and open the Java reflection module for those tests on Java 17. Existing
test cases and assertions are unchanged.

## How was this patch tested?

Validated rebased head `dc2784b985f00d6543dd3be6a563d181e7be1b8e`
against master `ef61df0920cec80f49c2142719f823dbf626c6ad` on Java 17
with Spark 4.2:

```sh
build/sbt -DsparkVersion=4.2 \
  "sparkV2/testOnly org.apache.spark.sql.delta.v2.tablemanager.CachedSnapshotManagerSuite org.apache.spark.sql.delta.v2.snapshot.unitycatalog.UCUtilsSuite" \
  "spark/Test/compile"
```

All 42 tests passed: 35 cache tests and 7 UC utility tests. Unified
compilation, including both Java changelog callers, and `git diff
--check` passed. The rebase preserves the seven-file contribution scope
and all existing assertions.

## Does this PR introduce _any_ user-facing changes?

No.

Signed-off-by: Hao Sun <86338940+HaoSun1993@users.noreply.github.com>
[AI-assisted]

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Centralize construction of the three CDC output columns in one helper.
Preserve nullable fields for the internal Parquet read schema and
non-null fields for changelog output, matching the existing contracts.

## How was this patch tested?

- `build/sbt "sparkV2/Test/javafmtCheck" "sparkV2/testOnly
org.apache.spark.sql.delta.v2.read.cdc.CDCSchemaContextTest"`

## Does this PR introduce _any_ user-facing changes?

No.

---------

Signed-off-by: tdas <tathagata.das1565@gmail.com>
Co-authored-by: omnigent <noreply@omnigent.ai>
…o#8175)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Use `Utils.tryWithSafeFinally` in Delta iterator helpers, streaming
source
cleanup, and eager log reads. When processing and closing both fail,
preserve
the processing exception and attach the close exception as suppressed. A
close
failure still propagates when processing succeeds.

Apply the same handling when constructing a wrapped iterator fails,
while keeping
successfully constructed lazy iterators open until their owner closes
them.

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

Added regression coverage in `LineClosableIteratorSuiteBase` for
processing
failures with and without close failures, close-only failures, wrapped
iterator
construction failures, and successful lazy iterator ownership.

`git diff --check`, `sparkV2/javafmt`, and `sparkV2/Test/javafmt`
passed.
Both Java formatter tasks completed without changing any files.

The two iterator suites passed against Spark 4.0: 32 tests, zero
failures.

```bash
build/sbt -DsparkVersion=4.0 \
  "spark/testOnly org.apache.spark.sql.delta.storage.InternalLineClosableIteratorSuite org.apache.spark.sql.delta.storage.PublicLineClosableIteratorSuite"
```

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

Yes. If processing and cleanup both fail, callers now receive the
original
processing exception with the cleanup exception suppressed, instead of
losing
the original failure. Successful processing behavior is unchanged.

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

---------

Signed-off-by: You Zhou <yz3883@columbia.edu>
[AI-assisted]

## What changes were proposed?

Define the Delta V2 enable modes as named constants in `DeltaSQLConf`
and reuse those constants in `DeltaV2Mode`. This removes duplicate
string literals without changing defaults, validation, case handling, or
connector-selection behavior.

## How was this patch tested?

- `sparkV2/compile`
- `spark/compile`
- `DeltaV2CDFSuite` (27 tests)
- `DeltaTableRefreshAutoModeSuite` and
`DeltaTableRefreshStrictModeSuite` (72 tests)

---------

Signed-off-by: tdas <tathagata.das1565@gmail.com>
…redicate hints (delta-io#8109)

#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (Sharing)

## Description

When a delta-format Delta Sharing query has one filter that can't be
converted to a json predicate (e.g. `RLIKE`/`LIKE`, or a large `IN`
list, which the optimizer turns into `InSet`), `OpConverter` throws. The
whole partition-filter or data-filter hint is then dropped, so the
sharing server can't skip any files and returns the full file list, even
though the other filters could have been used for skipping.

`DeltaSharingJsonPredicates.convert` now honors
`spark.delta.sharing.jsonPredicateHints.partialFilter.enabled`. That
conf, and partial conversion in `OpConverter`, shipped in
delta-sharing-client 1.4.1 (delta-io/delta-sharing#983), and the
parquet-format path (`RemoteDeltaFileIndex`) already uses them. When the
conf is on, `OpConverter` emits `UnsupportedOp` placeholders, and
`UnsupportedOpPruner` removes them before the hint is serialized:
- An AND keeps its supported children.
- An OR or NOT that contains an unsupported node is dropped whole.

So the hint still matches a superset of the files with matching rows,
and `UnsupportedOp` is never sent to the server. The conf applies to
both the partition and the data side of the hint. It defaults to
`false`, so behavior is unchanged unless it is enabled.

## How was this patch tested?

New `DeltaSharingFileIndexSuite` test "jsonPredicate partial filter
drops only unsupported sub-expressions". It pins the three hint confs
and asserts the exact json hint sent to the server:
- With the conf off, an unsupported `RLIKE` or a 21-value `IN` still
drops the whole data-filter side, as before.
- With the conf on:
- an unsupported AND conjunct (`RLIKE`, `InSet`, or a 21-value `IN`) is
dropped while the supported partition and data conjuncts are kept;
  - an OR containing an unsupported branch is dropped entirely;
  - a predicate with nothing supported sends no hint;
  - fully supported predicates produce the same hint as before.

Each check also asserts that exactly one file-listing call was made, so
a missing hint can't come from a cache hit.

## Does this PR introduce _any_ user-facing changes?

No. The behavior only changes when
`spark.delta.sharing.jsonPredicateHints.partialFilter.enabled` is set to
`true` (default `false`). With it on, queries with unsupported filters
can get server-side file skipping from their remaining filters.

This pull request and its description were written by Isaac.

---------

Signed-off-by: Abhijit Chakankar <abhijit.chakankar@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…or (delta-io#8173)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

`SingleCommit.getActionsIterator` returned the actions of a commit's log
file as written. For an inline manifest commit of a table with an
adaptive metadata tree, the `AddFile` and `RemoveFile` actions are
recorded in the tree, so the iterator did not return them. Callers that
process a commit's actions one at a time, without loading them all into
memory, need the actions that `getActions` returns.

- With `DeltaSQLConf.AMT_SINGLE_COMMIT_READ_ENABLED` on,
`getActionsIterator` returns the same actions as `getActions`, in the
same order: the log file's other actions, then the file actions
reconstructed from the tree. With the conf off, it returns the log file
as written.
- The new `AMTCheckpointProviderImpl.reconstructFileActionsIterator`
returns the actions of `reconstructFileActions`, in its order, and pulls
them to the driver one partition at a time. The iterator opens the tree
only after it has read the log file's actions. `rewind()` reads both
again.
- `getActions` and `getActionsIterator` share one path. A commit is an
inline manifest commit when its first action is a `CommitInfo` whose
`lastManifestCommit` is at the commit's own version
(`CommitInfo.isAMTInlineManifestCommit`). Its tree is located from its
`Checkpoint` action, and the read fails if that action is missing.
- The test helper `getJsonOnlyChanges` now reads the log files directly.

## How was this patch tested?

New tests in `AMTSingleCommitSuite`:
- `getActionsIterator` returns the same actions after `rewind()`.
- `getActions` and `getActionsIterator` return a manifest commit's
actions in the same order.
- `reconstructFileActionsIterator` returns the actions of
`reconstructFileActions` in order, with 7 shuffle partitions.
- `getActionsIterator` returns a log commit as written, and returns the
log file as written when single-commit read is off.

The existing `getChanges` test for an inline manifest commit also checks
`getActionsIterator`.

## Does this PR introduce _any_ user-facing changes?

No.
…io#8184)

[AI-assisted]

## Description

Adds a concurrency group to the Delta Spark workflow with
`cancel-in-progress: true`. PR runs are grouped by PR number, so a new
commit cancels the previous in-progress run for the same PR.

Non-PR runs—pushes to `master` or `branch-*`, and `workflow_dispatch`
runs—fall back to `github.run_id`, giving every run a unique concurrency
group. They are therefore never cancelled, preserving a full CI result
for every master commit and maintaining master regression detection.

## Testing

- Parsed `.github/workflows/spark_test.yaml` successfully with PyYAML.
- `git diff --check` passed.
- Confirmed the final diff against `delta/master` contains only the
concurrency comment and block.

---------

Co-authored-by: omnigent <noreply@omnigent.ai>
…lta-io#8180)

[AI-assisted]

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

Merge the Java and Scala map overload implementations of
`getPartitionRow` through a shared
helper. The overload-specific behavior remains unchanged, including the
Java map size assertion,
the Scala map missing-key handling, and lazy map access for empty
partition schemas.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

- `build/sbt "sparkV2/Test/javafmtCheck"`
- `build/sbt "sparkV2/testOnly
org.apache.spark.sql.delta.v2.utils.PartitionUtilsTest"`
  (23 test cases)

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

No.

---------

Signed-off-by: tdas <tathagata.das1565@gmail.com>
…8181)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Re-enable `incremental: commit file gap between versions,
failOnDataLoss=false succeeds`
in `DeltaV2SourceSuite` by moving it from `FailingTests` to
`PassingTests`.
Remove the obsolete comment that describes this case as unsupported.

This restores regression coverage for streaming across a gap between
commit versions
when `failOnDataLoss=false`.

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

`git diff --check` passed. Tests were not run locally because this
change only
re-enables an existing test; execution is left to the Spark CI workflow.

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

No. This change only updates test coverage.

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

Signed-off-by: You Zhou <yz3883@columbia.edu>
#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

The Delta protocol calls these tables catalog-managed, and the table
feature is `catalogManaged`. Comments in the Spark connector still say
"catalog-owned", "Catalog Owned" or "CatalogOwned tables". This PR
changes that prose to "catalog-managed".

## How was this patch tested?

Comment-only change.

## Does this PR introduce _any_ user-facing changes?

No.
…s definitions (delta-io#8158)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Stacked on delta-io#8151; review the last commit.

Renames the `CatalogOwnedTableFeature` and `CatalogOwnedTableUtils`
objects to `CatalogManagedTableFeature` and `CatalogManagedTableUtils`.
The feature name in the protocol stays `catalogManaged`.

The old names stay as aliases, in the `delta` package object and in a
new `coordinatedcommits` package object, so the callers keep compiling
until the PRs stacked on this one move them to the new names.
`UCDeltaCatalogClientImpl.scala` moves to the new name here.

## How was this patch tested?

Rename only, no new tests. Apart from the aliases and the line wrapping,
every changed token is an old name changed to the new one.

## Does this PR introduce _any_ user-facing changes?

No.
#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Record substantive Delta V2 snapshot loads, history requests, file-state
construction,
and table-manager cache lifecycle operations through the existing
operation hooks.
Normalize operation names under `delta.v2.` and attribute events to the
table path
without requiring a DeltaLog-backed provider. Cache timing uses a shared
elapsed-time helper.

Keep `KernelContext` focused on filesystem options and its LogStore.
Remove the unused
table-path constructor/factory argument and update the production and
test callers;
engine creation does not use that argument. Keep configuration
materialization scoped
to the active session with table options overlaid.

Path and payload construction for table-scoped events is guarded so
non-fatal
telemetry failures preserve the caller's control flow and original
exception.
Ordinary cache hits, snapshot freshness decisions, cleanup behavior, and
exceptions
retain their existing behavior.

This is the root of the operations stack: delta-io#8067, delta-io#8068. It is based
directly on master
and can be reviewed independently of the diagnostics stack (delta-io#8062,
delta-io#8066).
Design agreement is tracked in delta-io#8065 and remains pending.

## How was this patch tested?

Both `sparkV2/Test/compile` and `spark/Test/compile` passed. Six
affected suites
passed all 92 tests with zero failures or skips: KernelContextSuite (7),
CachedSnapshotManagerSuite (35), UCManagedTableSnapshotManagerSuite
(10),
DeltaV2SnapshotSuite (23), KernelSnapshotUtilsSuite (5), and
ApplyV2ReadOptionsSuite (12).

Style checks passed for all changed Scala files.

Whitespace checks pass. CI must rerun on the updated head.

## Does this PR introduce _any_ user-facing changes?

Yes. Delta V2 operation hooks gain consistent names and table
attribution.
Query results and the transaction-log format are unchanged.

---------

Signed-off-by: Zheng Hu <openinx@gmail.com>
…ts (delta-io#8182)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

The V2 scan Java tests cast the built scan to DeltaV2Scan in about 90
places. A failed cast gives a ClassCastException that does not say which
scan the builder built.

Add DeltaV2TestBase.asDeltaV2Scan, which checks the type and fails with
the class name of the scan, and use it in DeltaV2ScanTest,
DeltaV2ScanBuilderTest, and DeltaV2BatchTest. The test assertions do not
change.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

Existing tests

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->
No

Signed-off-by: Hari Kishore Chaparala <hchapara@uci.edu>
Co-authored-by: Isaac <no-reply@databricks.com>
…eltaV2MicroBatchStream (delta-io#8187)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

This PR starts to share the offset logic of the DSv1 streaming source
`DeltaSource` with the DSv2 streaming source `DeltaV2MicroBatchStream`.
Each PR moves one `DeltaSource` method, unchanged, into a shared trait,
and deletes the `DeltaV2MicroBatchStream` copy.

This PR adds the trait `DeltaSourceOffsetLogic`, which both sources mix
in, and moves `checkNonAdditiveSchemaChanges` into it.
`DeltaV2MicroBatchStream` now also runs the
`assert(!trackingMetadataChange)` check that `DeltaSource` has. Every
`DeltaV2MicroBatchStream` caller already skips the check while it tracks
metadata.

## How was this patch tested?

Existing tests. No test file changes.

## Does this PR introduce _any_ user-facing changes?

No.
#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other

## Description

`SnapshotManagementParallelListingSuite` registers its test coordinator
for `spark_catalog`. If another catalog is current, its unqualified
table name creates the table elsewhere and bypasses that coordinator.
Qualify the test table name and use an explicit catalog-qualified
identifier when opening the Delta log so the listing reconciliation
assertions exercise the intended coordinator.

## How was this patch tested?

- `git diff --check` passed.
- The local `spark/testOnly
org.apache.spark.sql.delta.SnapshotManagementParallelListingSuite` run
could not start because the sbt launcher download was unavailable. The
suite will run in CI.

## Does this PR introduce _any_ user-facing changes?

No.

Signed-off-by: Eric Tseng <52102337+EricTsengTy@users.noreply.github.com>
delta-io#8170)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Support querying versionless CDF for delta sharing clients with delta
format.
<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?
UTs
<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?
No
<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

---------

Co-authored-by: Omar Elhadidy <omar.elhadidy@databricks.com>
…a-io#8157)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Stacked on delta-io#8158; review the last commit.

Renames the `CatalogOwnedUsageLogs` object, which holds the op types of
catalog-managed tables, to `CatalogManagedUsageLogs`. The
`coordinatedcommits` package object keeps the old name as an alias. The
op type values, such as `delta.catalogOwned.commitToUcEtagRetry`, stay
the same.

## How was this patch tested?

Rename only, no new tests. Apart from the aliases and the line wrapping,
every changed token is an old name changed to the new one.

## Does this PR introduce _any_ user-facing changes?

No.
…builders (delta-io#8113)

[AI-assisted]

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Extracts the shared row-tracking file metadata lookup, missing-value
validation, and long conversion used by the V2 row ID and row commit
version value setter builders. Exception types and messages remain
unchanged.

## How was this patch tested?

- `build/sbt "sparkV2/Compile/javafmtCheckAll"`
- `build/sbt "sparkV2/testOnly
org.apache.spark.sql.delta.v2.read.metadata.MetadataStructReadFunctionTest
org.apache.spark.sql.delta.v2.read.metadata.MetadataStructSchemaContextTest
org.apache.spark.sql.delta.v2.read.metadata.MetadataValueSetterTest
org.apache.spark.sql.delta.v2.V2RowTrackingReadTest"`

## Does this PR introduce _any_ user-facing changes?

No.

---------

Signed-off-by: tdas <tathagata.das1565@gmail.com>
… have deletion vectors (delta-io#8190)

#### Which Delta project/connector is this regarding?

- [x] Spark

## Description

`DeletionVectorsTableFeature.validateDropInvariants` treated an unknown
DV count as zero.
`numDeletionVectorsOpt` is unknown when DVs are no longer writable and
the snapshot has no checksum
with the count, for example after a RESTORE commit. In that case DROP
FEATURE deletionVectors
succeeded while files still had deletion vectors.

This change checks the files themselves
(`DeletionVectorUtils.isTableDVFree`) when the count is
unknown.

## How was this patch tested?

Tested with a scenario that drops the feature after a RESTORE leaves the
DV count unknown while
files still have DVs.
Without this change DROP FEATURE succeeds; with it, DROP FEATURE fails
with
`DELTA_FEATURE_DROP_CONFLICT_REVALIDATION_FAIL`.

## Does this PR introduce _any_ user-facing changes?

No.

Co-authored-by: Isaac <no-reply@databricks.com>
delta-io#8194)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

A clustered table can be clustered by a field inside a struct, for
example `a.b`. DROP COLUMNS rejects a drop of a clustering column, but
it compares only whole names.

`DROP COLUMNS (a)` drops the struct `a`, and `a.b` with it. The check
misses that, because `"a"` is not `"a.b"`. The table is then clustered
by a column that is gone, and later writes, DESCRIBE DETAIL and OPTIMIZE
fail with `DELTA_CANNOT_RESOLVE_CLUSTERING_COLUMN`.

The check now also rejects a drop of a struct that holds a clustering
column: it matches when the name parts of a dropped column are a prefix
of the name parts of a clustering column.

## How was this patch tested?

New test in `ClusteredTableDDLSuite`: on a table clustered by `a.b`,
`DROP COLUMNS (a)`, `(A)` and `(a.b)` fail with
`DELTA_UNSUPPORTED_DROP_CLUSTERING_COLUMN`, and `DROP COLUMNS (a.c)`
succeeds and keeps `a.b` as the clustering column.

## Does this PR introduce _any_ user-facing changes?

Yes. `DROP COLUMNS` of a struct that holds a clustering column now fails
with `DELTA_UNSUPPORTED_DROP_CLUSTERING_COLUMN`. Before, it succeeded
and left the table clustered by a missing column.
)

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Make `DeltaDataFrameWriterV2Suite` easier to reuse with other table
providers while
preserving its existing Delta expectations:

- Extract partition assertions into overridable `assertPartitionedById`
and
  `assertUnpartitioned` helpers.
- Use the existing `createTableSQL` and `writeFormat` helpers in the
literal-string
  overwrite tests instead of hardcoded provider names.
- Remove an unused import.

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

## How was this patch tested?

`git apply --check` and `git diff --check` passed. The test suite has
not been run
in this checkout.

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

## Does this PR introduce _any_ user-facing changes?

No. This only refactors test scaffolding; the default assertions and
provider
remain unchanged.

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

Signed-off-by: Tejas Maire <36109575+tmaire111@users.noreply.github.com>
…elta-io#8121)

[AI-assisted]

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

`DeltaV2ScanUtils.newScanBuilder` was a pass-through to
`DeltaV2ScanBuilder.create`, which only derived the snapshot manager
from the table manager (`tableManager.snapshotManager(catalogTable)`)
and called the `DeltaV2ScanBuilder` constructor. Two factory layers did
one job.

This PR removes the extra layer:
- `DeltaV2ScanUtils.newScanBuilder` now constructs the
`DeltaV2ScanBuilder` directly, with the same arguments `create` used. It
stays the single entry point because `DeltaV2ScanBuilder` is
`private[read]` and its production caller, `DeltaV2Table`, lives in
another package. It also returns only the `ScanBuilder` interface.
- `DeltaV2ScanBuilder.create` is deleted.
- `DeltaV2ScanBuilderTest` now builds its scan builder through
`DeltaV2ScanUtils.newScanBuilder`, the same path `DeltaV2Table` uses.

This is a pure refactor with no behavior change.

## How was this patch tested?

Existing tests:
- `build/sbt "sparkV2/Test/javafmtCheck" "sparkV2/javafmtCheckAll"`
passes.
- `build/sbt "sparkV2/testOnly
org.apache.spark.sql.delta.v2.read.DeltaV2ScanBuilderTest
org.apache.spark.sql.delta.v2.read.DeltaV2ScanTest"` passes all 120 test
cases: 35 in `DeltaV2ScanBuilderTest` and 85 in `DeltaV2ScanTest`.

## Does this PR introduce _any_ user-facing changes?

No.

This pull request and its description were written by Isaac.

Signed-off-by: tdas <tathagata.das1565@gmail.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: omnigent <noreply@omnigent.ai>
…lta-io#8118)

[AI-assisted]

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Clarify that the limit-aware scan receives partition-only filters
because `build()` only sets `effectiveLimit` when there are no residual
data filters.

## How was this patch tested?

- `sparkV2/Compile/compileIncremental`
- `sparkV2/Compile/javafmtCheckAll`
- `sparkV2/Compile/scalafmtCheckAll` (blocked by pre-existing formatting
failures)

## Does this PR introduce _any_ user-facing changes?

No.

Signed-off-by: tdas <tathagata.das1565@gmail.com>
…ts (delta-io#8202)

## Summary

- Run coordinator retry and `commitLarge` failure tests against
catalog-managed Delta tables.
- Preserve the retry limits, exception checks, and failure telemetry
assertions.
- Remove a legacy property-based commit-directory test and mark four
coordinator cases as expected failures in the V2 interop suite.

## Testing

- `git diff --check`
- Focused sbt test could not start because the sbt launcher download
failed in the local checkout. Public CI will run the suite.

Signed-off-by: Eric Tseng <52102337+EricTsengTy@users.noreply.github.com>
…Coordinator* (delta-io#8153)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Stacked on delta-io#8157; review the last commit.

Renames `CatalogOwnedCommitCoordinatorBuilder` and
`CatalogOwnedCommitCoordinatorProvider` to
`CatalogManagedCommitCoordinatorBuilder` and
`CatalogManagedCommitCoordinatorProvider`. The `coordinatedcommits`
package object keeps both old names as aliases.

## How was this patch tested?

Rename only, no new tests. Apart from the aliases and the line wrapping,
every changed token is an old name changed to the new one.

## Does this PR introduce _any_ user-facing changes?

No.
…nabled (delta-io#8205)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Adds a required `isPreviewCompatible` parameter to
`AMTUtils.amtSupported` and `AMTUtils.amtEnabled`, so every caller
states whether it handles `adaptiveMetadata-preview` tables. With
`false`, preview tables don't count as AMT tables;
`adaptiveMetadata-dev` tables behave as before. New code paths, such as
`tightStats`, pass `false` so they don't run on preview tables, which
older clients still read and write.

Every existing caller passes `true`, so this PR doesn't change behavior.
The single-argument `amtEnabled(snapshot)` overload is removed; callers
use `amtEnabled(snapshot, isPreviewCompatible)`.

## How was this patch tested?

`AMTUtilsSuite` checks every feature and AMT property case with the flag
set to `true` and to `false`. `AdaptiveMetadataTableFeatureSuite` checks
the snapshot form on real tables: with `false`, only
`adaptiveMetadata-dev` tables count as AMT tables.

## Does this PR introduce _any_ user-facing changes?

No.
linzhou-db and others added 15 commits November 17, 2026 17:44
…ta-io#8206)

#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (Delta Sharing)

## Description

Removes the internal conf
`spark.sql.delta.sharing.enableAutoResolveForCdf`
(`DeltaSQLConf.DELTA_SHARING_ENABLE_AUTO_RESOLVE_FOR_CDF`). A Delta
Sharing batch CDF read without
an explicit `responseFormat` now always auto-resolves its response
format.

- `DeltaSharingDataSource.createRelation`: every batch read without a
`responseFormat`, snapshot
or CDF, goes to `autoResolveBaseRelation`. That method calls
getMetadata, pinned to the end of
the CDF range (or the latest version when the range has no end), and
takes the parquet or delta
path that the server answers with. Before this change, a CDF read with
the conf off (its
default) fell back to the legacy parquet path. The
`shouldAutoResolveCDF` helper is removed.
  Reads with an explicit `responseFormat` are unchanged.
- `DeltaSQLConf`: the conf is removed.
- `DeltaSharingDataSourceDeltaSuite`: the two tests parameterized on the
conf are replaced by
always-on versions. One reads delta-format CDF across version and
timestamp bounds. The other
reads a parquet-format share and checks that it resolves to the parquet
path (the parquet CDF
  reader's `LongType` `_commit_timestamp`).

The conf was added so the CDF auto-resolve path could be rolled out
gradually. That rollout is
done, so the conf is no longer needed.

## How was this patch tested?

Updated the tests in `DeltaSharingDataSourceDeltaSuite` as described
above.

- `build/sbt "spark/scalastyle" "sharing/scalastyle"
"sharing/Test/scalastyle" "sharing/testOnly io.delta.sharing.spark.*"`
Result: scalastyle reports 0 errors and 0 warnings. `sharing/testOnly`:
130 passed, 2 failed.
The same 2 tests fail with this command on unmodified master
(3b1bb78):
`DeltaSharingDataSource able to auto resolve responseFormat` fails only
when run after
`DeltaFormatSharingSourceSuite` in the same JVM (the global
`requestedFormat` map collects
`shared_parquet_table*` entries from that suite) and passes when its
suite runs alone.
`versionless CDF refresh preserves timestamp bounds and isolates
queries`
  (`DeltaSharingCDFUtilsSuite`) hits a 20s `eventually` timeout.

## Does this PR introduce _any_ user-facing changes?

Yes. A Delta Sharing batch CDF read without a `responseFormat` option
(for example
`spark.read.format("deltaSharing").option("readChangeFeed",
"true").load(path)` or
`table_changes`) now asks the server for the table's format and reads
through the delta-format
path when the server answers delta. Before, by default it always used
the parquet path. The
removed conf was internal.

This pull request and its description were written by Isaac.

Signed-off-by: lin.zhou <lin.zhou@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…nseFormat (delta-io#8207)

#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [x] Other (Delta Sharing)

## Description

Removes the internal conf
`spark.sql.delta.sharing.cdfStreamingAutoResolveResponseFormat`

(`DeltaSQLConf.DELTA_SHARING_CDF_STREAMING_AUTO_RESOLVE_RESPONSE_FORMAT`).
Delta Sharing streaming
CDF (`readChangeFeed=true`) now always auto-resolves its response
format.

- `DeltaSharingDataSource.autoResolveStreamingSource`: a CDF stream
always uses getMetadata. When
`responseFormat` is not set, it takes the format the server answers
with, and an explicit
`responseFormat=delta` goes to the delta-format source. Non-CDF
streaming is still gated by
  `spark.sql.delta.sharing.streamingAutoResolveResponseFormat`.
- `DeltaFormatSharingSource.isAutoResolveEnabled`: always true for CDF
streams, so the
delta-format source accepts a legacy `DeltaSharingSourceOffset`
checkpoint left by an earlier
  parquet CDF stream and picks the matching file-id hash.
- `DeltaSQLConf`: the conf is removed.
- `DeltaFormatSharingSourceSuite`: the `forceToDeltaSourceOffset`
flag-routing grid now covers
CDF auto-resolve with the non-CDF conf both on and off, and non-CDF with
the non-CDF conf on
  and off.

The conf was added so CDF streaming auto-resolve could be rolled out
separately from non-CDF
streaming. That rollout is done, so the conf is no longer needed.

## How was this patch tested?

Updated the routing-grid test in `DeltaFormatSharingSourceSuite` as
described above.

- `build/sbt "spark/scalastyle" "sharing/scalastyle"
"sharing/Test/scalastyle" "sharing/testOnly io.delta.sharing.spark.*"`
Result: scalastyle reports 0 errors and 0 warnings. `sharing/testOnly`:
130 passed, 2 failed.
The same 2 tests fail with this command on unmodified master
(3b1bb78):
`DeltaSharingDataSource able to auto resolve responseFormat` fails only
when run after
`DeltaFormatSharingSourceSuite` in the same JVM (the global
`requestedFormat` map collects
`shared_parquet_table*` entries from that suite) and passes when its
suite runs alone.
`versionless CDF refresh preserves timestamp bounds and isolates
queries`
  (`DeltaSharingCDFUtilsSuite`) hits a 20s `eventually` timeout.

## Does this PR introduce _any_ user-facing changes?

Yes. A Delta Sharing streaming CDF read now always asks the server for
the table's format. If
`responseFormat` is not set and the server answers delta, the stream
uses the delta-format source
even when `spark.sql.delta.sharing.streamingAutoResolveResponseFormat`
is off. Before, by default
it used the parquet source. A CDF stream that restarts with
`responseFormat=delta` from a legacy
parquet CDF checkpoint now resumes from that checkpoint instead of
failing to parse it. The
removed conf was internal.

This pull request and its description were written by Isaac.

Signed-off-by: lin.zhou <lin.zhou@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
…#8208)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

The V4 spec lets a leaf manifest hold both live files and tombstones,
and other engines write such leaves, for example Iceberg after a delete
and insert. The incremental AMT writer rejected such a leaf when it
carried the leaf forward, so every later write on the table failed with:

```
[INTERNAL_ERROR] ManifestInfo representing a leaf's initial state must not contain both live files and tombstones.
```

Readers already skip a carried leaf's tombstones by entry status, so
this writer check was the only thing in the way. This PR:

- Removes the check and its alert.
- Records a non-fatal usage log instead, once per incremental write that
carries mixed leaves, with how many it carried.
- Picks a carried leaf's tracking from its live files only. Before, any
leaf with a tombstone went straight to DELETED, which would have dropped
a mixed leaf's live files.
- Assumes a manifest deletion vector masks live files only, as the path
that adds new masking already does. A leaf becomes DELETED once its
deletion vector covers all of its live files. A deletion vector that
covers more entries than the leaf has live files fails the write with
the `amt.commit.mdvExceedsLiveFiles` alert instead of guessing which
entry is still live.

The V4 spec wants a leaf MODIFIED in the commit that masks its last live
file and DELETED in the next commit. The writer still marks it DELETED
right away, as before.

## How was this patch tested?

New tests in `AMTIncrementalWriteInvariantSuite`:

- Unit tests for `carryForwardOneLeaf` with a mixed leaf (EXISTING,
MODIFIED and DELETED outcomes), a mixed leaf an earlier commit fully or
partly masked, a manifest deletion vector that masks more entries than
the live files, and live-only and tombstone-only leaves for comparison.
- An end-to-end test that swaps one leaf of a real AMT table for a copy
with an extra DELETED entry, then runs INSERTs and DELETEs on top. At
each step it checks the leaf's tracking, the change feed (the tombstone
never shows up as a remove), and the table's rows. It also checks that
the first write records the mixed-leaf usage log once. Without the fix,
the first write fails with the error above.

## Does this PR introduce _any_ user-facing changes?

No.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
…#8172)

## Description

When a snapshot discovers an AMT checkpoint from storage, the columnar
provider can keep the root's rows in memory and reuse them. A warm
commit does not go through discovery: it builds the next provider from
`AMTWriteResult`, which only carried the leaf pointers. So the next
incremental AMT write had to read the root Parquet file again.

This PR carries the complete `AMTRootInfo`, leaf pointers plus any
cached root rows, through `AMTWriteResult`. It keeps the rows the writer
produced when the root is below the root-cache threshold:

- Incremental writes and multi-leaf full writes keep the rows built on
the driver to write the root.
- When a full write produces a single leaf that is promoted to the root,
no rows are kept, and the next commit reads that root from storage.

The writer and discovery use the same check,
`AMTUtils.shouldCacheRootRows`: columnar state reconstruction is enabled
and the root has fewer entries than `amt.rootCache.maxNumEntries`. The
cached rows are copied into unsafe rows so they don't depend on the
writer's buffers.

Turning the written rows into a local relation now applies
`ConvertToLocalRelation` until the plan stops changing
(`AMTUtils.convertToLocalRelationFully`). When the rule visits the plan
top-down, one application folds only the projection directly above the
local relation, which made the writer silently skip caching. The writer
now asserts that the rows fold to a local relation. Decoding leaf
pointers from a cached root uses the same helper.

No extra Spark scan is added. Large roots, and the case where root
caching is off (`amt.rootCache.maxNumEntries` = 0), keep reading the
root from storage as before.

This PR also raises the default of `amt.rootCache.maxNumEntries` from
12000 to 50000.

## How was this patch tested?

New tests in `AMTSnapshotDiscoverySuite`:
- After a warm commit, the columnar provider reuses the root rows the
writer produced.
- The writer does not keep root rows when the root is at or above the
threshold, or when caching is off.
- Discovering a cached root decodes its leaf pointers without launching
another Spark job.

`LastCheckpointInfoSuite` is updated for the new `AMTWriteResult` shape.

## Does this PR introduce _any_ user-facing changes?

No.

Co-authored-by: Jinhua Song <jinhua.song@databricks.com>
#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Stacked on delta-io#8153; review the last commit.

Moves callers of `CatalogOwnedTableFeature` to the new name
`CatalogManagedTableFeature`. Each change is the name only.

## How was this patch tested?

Rename only, no new tests. Apart from the aliases and the line wrapping,
every changed token is an old name changed to the new one.

## Does this PR introduce _any_ user-facing changes?

No.
…ta-io#8203)

#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [x] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

When a Flink job is resubmitted from a checkpoint with a new job ID, the
committer uses the new ID to replay work created by the old job. Delta
treats the replay as a new transaction. For example, if OPTIMIZE has
replaced an old file, replay can add that file back to the active table.
If VACUUM has deleted it, the table can then reference a missing file.

Use the job ID saved in each committable to identify its transaction.
Group work by that saved job ID before sorting it by checkpoint number,
so recovered work and new work stay separate even when their checkpoint
numbers match. The checkpoint serialization format stays unchanged. The
committer builder no longer needs a job ID; `withJobId()` remains as a
deprecated no-op for compatibility.

Upgrade caveat: if an older connector first committed checkpoint work
under a replacement job ID, and there is no transaction record for its
original job ID, restoring that checkpoint after upgrading can replay it
once more. This change does not repair transaction history written by
older versions.

## How was this patch tested?

local and CI tests.

## Does this PR introduce _any_ user-facing changes?

Restores the intended behavior when resubmitting a job from a checkpoint
with a new job ID: already committed work is skipped, files removed by
later transactions stay removed, and new work still commits.

---------

Signed-off-by: Timothy Wang <31230953+timothyw553@users.noreply.github.com>
…8215)

[AI-assisted]

<!--
Thanks for sending a pull request!  Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]
Your PR title ...'.
  3. Be sure to keep the PR description updated to reflect all changes.
  4. Please write your PR title to summarize what this PR proposes.
5. If possible, provide a concise example to reproduce the issue for a
faster review.
6. If applicable, include the corresponding issue number in the PR title
and link it in the body.
-->

#### Which Delta project/connector is this regarding?
<!--
Please add the component selected below to the beginning of the pull
request title
For example: [Spark] Title of my pull request
-->

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

<!--
- Describe what this PR changes.
- Describe why we need the change.
 
If this PR resolves an issue be sure to include "Resolves #XXX" to
correctly link and close the issue upon merge.
-->

This test-only follow-up to delta-io#8105 exercises the production metadata
schema conversion path with
the converter's former test inputs. It also adds a smoke test for
UTF8_BINARY string schemas,
including array and nested-struct shapes.

## How was this patch tested?

<!--
If tests were added, say they were added here. Please make sure to test
the changes thoroughly including negative and positive cases if
possible.
If the changes were tested in any way other than unit tests, please
clarify how you tested step by step (ideally copy and paste-able, so
that other reviewers can test and check, and descendants can verify in
the future).
If the changes were not tested, please explain why.
-->

```shell
build/sbt "sparkV2/Test/javafmtCheck" "sparkV2/javafmtCheck"
build/sbt "sparkV2/testOnly org.apache.spark.sql.delta.v2.utils.SchemaUtilsTest"
```

`SchemaUtilsTest` passes all 44 test cases.

## Does this PR introduce _any_ user-facing changes?

<!--
If yes, please clarify the previous behavior and the change this PR
proposes - provide the console output, description and/or an example to
show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change
compared to the released Delta Lake versions or within the unreleased
branches such as master.
If no, write 'No'.
-->

No.

---------

Signed-off-by: tdas <tathagata.das1565@gmail.com>
Co-authored-by: omnigent <noreply@omnigent.ai>
#### Which Delta project/connector is this regarding?

- [ ] Spark
- [ ] Standalone
- [ ] Flink
- [x] Kernel
- [ ] Other (fill in here)

## Description

Java Kernel searches for checkpoints in fixed 1,000-version batches. On
tables with frequent multipart checkpoints, a batch can contain many
objects and make snapshot loading expensive.

This PR adds `SnapshotBuilder.withCheckpointSearchBatchSize(int)` so
callers can choose a positive batch width per snapshot. Kernel uses that
width for both the initial search and stale-checkpoint fallback, and
continues into older batches until it finds a complete checkpoint. The
default remains 1,000 versions. The internal Checkpointer and full
SnapshotManager calls require the width explicitly. Checkpoint search
also closes each listing iterator and stops after checking the
zero-start batch once.

## How was this patch tested?

Added tests for a width of 5 across older batches and stale-checkpoint
fallback, nonpositive width rejection, missing and incomplete
checkpoints at the zero-start batch, and listing-iterator closure across
batches and on return.

From the repository root with JDK 17:

```bash
sbt --java-home /usr/lib/jvm/java-17-openjdk-amd64 -batch -no-colors 'kernelApi/testOnly io.delta.kernel.*'
```

Passed: 2,985 tests, 0 failures, 0 errors.

## Does this PR introduce _any_ user-facing changes?

Yes. `SnapshotBuilder` gains `withCheckpointSearchBatchSize(int)`.
Previously, checkpoint discovery always searched 1,000 versions per
batch. Callers can now set a positive width for a snapshot; callers that
do not set it retain the 1,000-version default.

---------

Signed-off-by: Thinh Bui <26578143+thinh2@users.noreply.github.com>
…elta-io#8218)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Match partition columns by their already-resolved names in
`DeltaV2ScanBuilder.pruneColumns`,
instead of lowercasing both sides. This preserves a metadata field named
`__metadata` beside a
partition column named `__Metadata`, while still excluding the partition
column from the data-read
schema. CDC column filtering remains case-insensitive.

## How was this patch tested?

Added a regression test using a Delta table with a stored `_metadata`
column and a `__Metadata`
partition column. The test passes a resolved `__metadata` metadata field
directly to the scan builder
and verifies that pruning preserves it under both case-sensitivity
settings.

Local test execution is deferred to CI. `git diff --check` passed.

## Does this PR introduce _any_ user-facing changes?

Yes. V2 column pruning preserves resolved metadata fields whose names
differ from partition-column
names only by case, instead of discarding them.

Signed-off-by: You Zhou <yz3883@columbia.edu>
…io#8219)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other

## Description

V2 streaming reads commit JSON through LogStore. Validate each protocol
action with the shared `Protocol.protocolRead` check before returning
the commit's files or collecting metadata and protocol actions. This
rejects an unsupported historical protocol even when the latest snapshot
has a supported protocol.

Pass the stream's Spark session and read snapshot to the action
collectors. Add coverage for unsupported reader versions and reader
features, supported writer-only features, lazy validation, and reader
cleanup. Enable the existing historical-protocol streaming regression
tests.

## How was this patch tested?

- Four focused suites passed with Java 17 and Spark 4.2: **325 tests
passed, 0 failures** (`StreamingHelperTest`,
`DeltaV2MicroBatchStreamTest`, `MetadataEvolutionHandlerTest`, and
`DeltaV2TableManagerImplSuite`).
- `git diff --check`
- Java formatting checks with Java 17 and Spark 4.2:

```sh
./build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 -DsparkVersion=4.2 \
  'sparkV2/javafmtCheckAll'
```

## Does this PR introduce _any_ user-facing changes?

On master, V2 streams now reject unsupported historical reader protocols
with the standard Delta protocol-read errors before emitting files from
that commit. Unknown writer-only features remain readable. No public API
or table format changes.

Signed-off-by: Hao Sun <86338940+HaoSun1993@users.noreply.github.com>
…elta-io#8214)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

The stats JSON of AMT entries is now built from a single stats struct,
assembled out of plain field accesses into the typed `content_stats`
column, instead of rebuilding the nested shape path by path. Statistics
are matched to Delta's stats schema by physical path, a matched
statistic of a widened type is cast, and statistics the manifests do not
persist read as null. The JSON itself is unchanged, including leaving
out min/max groups that have no persisted values rather than writing
`{}`.

## How was this patch tested?

Existing AMT stats and state reconstruction suites, plus a new
`AMTContentStatsSuite` test for a table where no column has persisted
bounds.

## Does this PR introduce _any_ user-facing changes?

No

Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
…AMT stats (delta-io#8223)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

Add an `AMTFieldSpecSuite` test that drops a column and re-adds a column
with the same name, which gets a new column mapping id. In each AMT
checkpoint scenario, it checks that:
- the tree written before the drop holds min/max/null-count stats for
the dropped column,
- the files written before the drop carry no stats for the re-added
column, with Parquet field-id reads on and off, and the new file keeps
its stats,
- an `IS NULL` filter on the re-added column returns every row of the
old files.

## How was this patch tested?

Test-only change.

## Does this PR introduce _any_ user-facing changes?

No.
#### Which Delta project/connector is this regarding?

- [x] Spark

## Description

Sharing checkpointed setup across compatible AMT back-reference checks
avoids repeated fixture construction while keeping clone targets
isolated.

| Pre-export source-suite workload and timed scope | Before | After |
Benefit |
| --- | ---: | ---: | ---: |
| Complete five-shard `AMTBackReferenceSuite`, mean command wall time |
463.26 s | 430.72 s | 7.0% lower |

Two alternating pre-export runs per source revision compared
`a3ac97a194a85815a3ec9572939e882e4f053aa0` with
`19a2cde8bfd72b4087188595489f482eed6a2455`, including startup, fixtures
and teardown. Later source cleanups and this public head were not timed.
CPU and CI savings are unmeasured.

## How was this patch tested?

The 75-case pre-export source suite retains all original obligations
across three checkpoint modes, with deliberate back-reference faults and
alternate-order checks. Rejected transactions preserve table version and
live references. The generated patch was applied to public master. The
cleanup preserves the shared helper assertions and was checked in all
three Runtime checkpoint modes. The transformed public suite has not
been run locally. All 53 public CI checks pass on
`a9d933322ec58ebc3d3a39fa8dc397df711af3ae`.

## Does this PR introduce any user-facing changes?

No.

<details>
<summary>Agent attribution</summary>

Agent: Costly Tests - Orchestrator | Session:
01a1e505-448e-7d81-b315-7ef768b0399c
Agent: Costly Tests - SQL | Session:
01a1e523-4d27-73b1-b47b-a8e576721388
</details>

---------

Signed-off-by: Cyrus Najmabadi <cyrus.najmabadi@databricks.com>
Signed-off-by: cyrus.najmabadi <cyrus.najmabadi@databricks.com>
Co-authored-by: Cyrus Najmabadi <cyrus.najmabadi@databricks.com>
#### Which Delta project/connector is this regarding?

- [x] Spark

## Description

Sharing checkpoints across compatible AMT snapshot checks avoids
repeated fixture construction while keeping mutable histories isolated.

| Pre-export source-suite workload and timed scope | Before | After |
Benefit |
| --- | ---: | ---: | ---: |
| Complete five-shard `AMTSnapshotSuite`, mean first test-action start
through last test-action finish | 371.740 s | 352.347 s | 5.22% lower |

Two alternating pre-export runs per source revision compared
`0af7c6e203e660d0073f25e2b0c261ecfa0e527d` with
`408de8dba1118392ce5e6018424760af41cb7fa3`, including startup, fixtures
and teardown. Later source cleanups and this public head were not timed.
CPU and CI savings are unmeasured.

## How was this patch tested?

The 87-case pre-export source suite retains all 99 original obligations
across three checkpoint modes, with alternate-order and deliberate
DELETE/manifest-deletion-vector faults. The final generated patch was
applied to public master, and the resulting file matches the reviewed
output blob. The transformed public suite has not been run locally.
Public CI passes on `1849c03b526045aee1b42e7dd43c467ca159fcf3`.

## Does this PR introduce any user-facing changes?

No.

<details>
<summary>Agent attribution</summary>

Agent: Costly Tests - Orchestrator | Session:
01a1e505-448e-7d81-b315-7ef768b0399c
</details>

Signed-off-by: Cyrus Najmabadi <cyrus.najmabadi@databricks.com>
Co-authored-by: Cyrus Najmabadi <cyrus.najmabadi@databricks.com>
Adds a metadata-only command that removes the partitioning of a Delta
table without rewriting data files:

  ALTER TABLE t REPLACE PARTITIONED BY WITH CLUSTER BY
  ALTER TABLE t REPLACE PARTITIONED BY WITH CLUSTER BY (c1, c2)
  ALTER TABLE t REPLACE PARTITIONED BY WITH CLUSTER BY NONE

Without columns, the partition columns become the clustering columns;
with NONE the table becomes a plain unpartitioned table.

The command verifies that every data file physically stores the
partition columns (written when delta.writePartitionColumnsToParquet is
enabled), then re-adds all files in a single commitLarge commit with
empty partitionValues and dataChange=false. The stats of each file are
extended with min/max/nullCount for the former partition columns, and
delta.dataSkippingStatsColumns is set when they would otherwise lose
stats, so data skipping on them keeps working. The existing stats are
kept exactly. Tables with IcebergCompat, UniForm or adaptive metadata
enabled are rejected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Eunjin Song <sezruby@gmail.com>
@sezruby
sezruby force-pushed the replace-partitioned-by-cluster-by branch from 70a746e to 685eec2 Compare November 18, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.