Repository navigation
Conversation
1 of 5 tasks
…hema in the V2 connector (delta-io#8105) [AI-assisted] #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description The V2 connector's `SchemaUtils` has a Kernel-to-Spark schema converter: `convertKernelSchemaToSparkSchema`, `convertKernelDataTypeToSparkDataType`, `convertKernelFieldMetadataToSparkMetadata`, and their private helpers. Main code never calls them. Table schemas are built with `convertKernelMetadataToSparkSchema`, which parses the log's schema string with `DataType.fromJson`. Since delta-io#7942, the converter also rejects collated strings. This PR: - removes the unused converter and its helpers. The Spark-to-Kernel direction is unchanged. - removes the `SchemaUtilsTest` cases that only covered the removed code, and the Kernel-to-Spark halves of the round-trip checks. - changes `PartitionUtilsTest` to get a snapshot's Spark schema through `convertKernelMetadataToSparkSchema`, as the connector does. ## How was this patch tested? `build/sbt "sparkV2/Test/javafmtCheck" "sparkV2/javafmtCheck" "sparkV2/testOnly org.apache.spark.sql.delta.v2.utils.SchemaUtilsTest org.apache.spark.sql.delta.v2.utils.PartitionUtilsTest"`: formatting checks are clean, and 65 JUnit cases pass (`SchemaUtilsTest` 42, `PartitionUtilsTest` 23). ## Does this PR introduce _any_ user-facing changes? No. --------- Signed-off-by: tdas <tathagata.das1565@gmail.com>
…-io#8171) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Before writing data files, Delta makes the query's output attributes nullable (`TransactionalWrite.makeOutputNullable`), so that a query that under-reports nullability can't produce unreadable Parquet files. The helper it uses, `SchemaUtils.typeAsNullable`, skips primitive map values and the elements of nested arrays and maps. When such a field holds NULLs that the query reports as NOT NULL (e.g. from `lag`/`lead` with a NULL default), they land in a required Parquet field: the commit succeeds, but reading the table fails. This affects CTAS, RTAS, INSERT INTO, INSERT OVERWRITE, DataFrame writes and MERGE INSERT. This PR adds `SchemaUtils.typeAsFullyNullable`, which makes every nested field nullable, and uses it in `makeOutputNullable`. Only the Parquet schemas of the data files change; the table schema in the Delta log was already fully nullable. `typeAsNullable` is unchanged, because the DDL NOT NULL validation (`removeUnenforceableNotNullConstraints`) also uses it, and making it fully recursive would reject table schemas that are accepted today. Its doc now says what it leaves as it is. ## How was this patch tested? New tests in `SchemaUtilsSuite` cover what `typeAsNullable` and `typeAsFullyNullable` return for each kind of nested type. New tests in `TransactionalWriteSuite` write a query whose `lag`/`lead` defaults put NULLs into a primitive map value, nested array elements and values nested in map values, which the query reports as NOT NULL: - For each write (CTAS, RTAS, CREATE OR REPLACE TABLE AS SELECT, INSERT INTO, INSERT OVERWRITE, DataFrame save/append/overwrite, DataFrameWriterV2 append, MERGE INSERT, INSERT BY NAME, MERGE UPDATE, UPDATE), the NULLs read back correctly. - A check that the query reports these fields as NOT NULL but returns NULLs there, so the tests keep exercising the bug. ## Does this PR introduce _any_ user-facing changes? Yes, a bug fix: writes whose query reports a primitive map value or a nested array/map element as NOT NULL, but returns NULLs there, used to commit Parquet files that couldn't be read afterwards. They now produce readable files, and the NULLs read back.
…implementation (delta-io#8149) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Reconstruct the change files of an AMT commit per state reconstruction implementation. The change file reconstruction decodes each manifest row into an `AMTDataEntryExtended` and converts it with `DataEntryResolver`, while the columnar state reconstruction builds the same `AddFile`s as columns. This gives the columnar implementation a change file read built from those columns, so it has a single `AddFile` derivation. The per-row implementation keeps its read and stays the default and the fallback. Review it commit by commit in the stacked history this was exported from; the change is: - `trackingDerivedChangeActions` and `mdvDerivedChangeActions` are declared on `AMTCheckpointProviderImpl`. `PerRowAMTCheckpointProvider` keeps the existing per-row code. `ColumnarAMTCheckpointProvider` builds them from its `AddFile` columns. - Columnar change files: a live entry becomes its `add` column, stamped with the commit-level `dataChange` and without a back reference. Any other entry becomes a `RemoveFile` built from that `add` the way `removeWithTimestamp` builds it. `removeFileColumn` maps every `RemoveFile` field explicitly, so a new field fails the read until it is mapped. - `rootDataEntries` and `leafDataEntries` are shared by the snapshot reads, the MDV read and the tracking-derived change file read. `leafEntriesWithRowIdPrefix` is replaced by `leafDataEntries`. - `PerRowAMTCheckpointProvider` and `ColumnarAMTCheckpointProvider` move into their own files; `AMTCheckpointProvider` is no longer sealed. ## How was this patch tested? `AMTCommitReconstructionSuite` runs every test with both implementations and compares the reconstructed change files with the committed ones (adds, removes and MDV-derived removes). It also asserts the two implementations produce identical actions field by field, and that each reconstructed `RemoveFile` equals `AddFile.removeWithTimestamp` of the previous `AddFile`. `AMTAddFileFieldCoverageSuite` pins the `RemoveFile` fields and calls `removeFileColumn` for each. `AMTAddFileReconstructionSuite` checks that both implementations reject a leaf entry read from a manifest that is not one of its parent's leaves. ## Does this PR introduce _any_ user-facing changes? No. Co-authored-by: Isaac <no-reply@databricks.com>
#### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other ## Description Convert `SnapshotManagerFactory` from a Java utility class to a Scala object and accept `Option[CatalogTable]`. Scala callers can pass catalog metadata directly, while Java callers use Scala `Option`. Preserve catalog selection, lookup timing, exceptions, and lazy path-based fallback. Keep the factory scoped to the v2 connector. Adapt the existing cache tests to scoped Scala-object spying, add the required test dependency, and open the Java reflection module for those tests on Java 17. Existing test cases and assertions are unchanged. ## How was this patch tested? Validated rebased head `dc2784b985f00d6543dd3be6a563d181e7be1b8e` against master `ef61df0920cec80f49c2142719f823dbf626c6ad` on Java 17 with Spark 4.2: ```sh build/sbt -DsparkVersion=4.2 \ "sparkV2/testOnly org.apache.spark.sql.delta.v2.tablemanager.CachedSnapshotManagerSuite org.apache.spark.sql.delta.v2.snapshot.unitycatalog.UCUtilsSuite" \ "spark/Test/compile" ``` All 42 tests passed: 35 cache tests and 7 UC utility tests. Unified compilation, including both Java changelog callers, and `git diff --check` passed. The rebase preserves the seven-file contribution scope and all existing assertions. ## Does this PR introduce _any_ user-facing changes? No. Signed-off-by: Hao Sun <86338940+HaoSun1993@users.noreply.github.com>
[AI-assisted] #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Centralize construction of the three CDC output columns in one helper. Preserve nullable fields for the internal Parquet read schema and non-null fields for changelog output, matching the existing contracts. ## How was this patch tested? - `build/sbt "sparkV2/Test/javafmtCheck" "sparkV2/testOnly org.apache.spark.sql.delta.v2.read.cdc.CDCSchemaContextTest"` ## Does this PR introduce _any_ user-facing changes? No. --------- Signed-off-by: tdas <tathagata.das1565@gmail.com> Co-authored-by: omnigent <noreply@omnigent.ai>
…o#8175) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Use `Utils.tryWithSafeFinally` in Delta iterator helpers, streaming source cleanup, and eager log reads. When processing and closing both fail, preserve the processing exception and attach the close exception as suppressed. A close failure still propagates when processing succeeds. Apply the same handling when constructing a wrapped iterator fails, while keeping successfully constructed lazy iterators open until their owner closes them. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? Added regression coverage in `LineClosableIteratorSuiteBase` for processing failures with and without close failures, close-only failures, wrapped iterator construction failures, and successful lazy iterator ownership. `git diff --check`, `sparkV2/javafmt`, and `sparkV2/Test/javafmt` passed. Both Java formatter tasks completed without changing any files. The two iterator suites passed against Spark 4.0: 32 tests, zero failures. ```bash build/sbt -DsparkVersion=4.0 \ "spark/testOnly org.apache.spark.sql.delta.storage.InternalLineClosableIteratorSuite org.apache.spark.sql.delta.storage.PublicLineClosableIteratorSuite" ``` <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? Yes. If processing and cleanup both fail, callers now receive the original processing exception with the cleanup exception suppressed, instead of losing the original failure. Successful processing behavior is unchanged. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> --------- Signed-off-by: You Zhou <yz3883@columbia.edu>
[AI-assisted] ## What changes were proposed? Define the Delta V2 enable modes as named constants in `DeltaSQLConf` and reuse those constants in `DeltaV2Mode`. This removes duplicate string literals without changing defaults, validation, case handling, or connector-selection behavior. ## How was this patch tested? - `sparkV2/compile` - `spark/compile` - `DeltaV2CDFSuite` (27 tests) - `DeltaTableRefreshAutoModeSuite` and `DeltaTableRefreshStrictModeSuite` (72 tests) --------- Signed-off-by: tdas <tathagata.das1565@gmail.com>
…redicate hints (delta-io#8109) #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (Sharing) ## Description When a delta-format Delta Sharing query has one filter that can't be converted to a json predicate (e.g. `RLIKE`/`LIKE`, or a large `IN` list, which the optimizer turns into `InSet`), `OpConverter` throws. The whole partition-filter or data-filter hint is then dropped, so the sharing server can't skip any files and returns the full file list, even though the other filters could have been used for skipping. `DeltaSharingJsonPredicates.convert` now honors `spark.delta.sharing.jsonPredicateHints.partialFilter.enabled`. That conf, and partial conversion in `OpConverter`, shipped in delta-sharing-client 1.4.1 (delta-io/delta-sharing#983), and the parquet-format path (`RemoteDeltaFileIndex`) already uses them. When the conf is on, `OpConverter` emits `UnsupportedOp` placeholders, and `UnsupportedOpPruner` removes them before the hint is serialized: - An AND keeps its supported children. - An OR or NOT that contains an unsupported node is dropped whole. So the hint still matches a superset of the files with matching rows, and `UnsupportedOp` is never sent to the server. The conf applies to both the partition and the data side of the hint. It defaults to `false`, so behavior is unchanged unless it is enabled. ## How was this patch tested? New `DeltaSharingFileIndexSuite` test "jsonPredicate partial filter drops only unsupported sub-expressions". It pins the three hint confs and asserts the exact json hint sent to the server: - With the conf off, an unsupported `RLIKE` or a 21-value `IN` still drops the whole data-filter side, as before. - With the conf on: - an unsupported AND conjunct (`RLIKE`, `InSet`, or a 21-value `IN`) is dropped while the supported partition and data conjuncts are kept; - an OR containing an unsupported branch is dropped entirely; - a predicate with nothing supported sends no hint; - fully supported predicates produce the same hint as before. Each check also asserts that exactly one file-listing call was made, so a missing hint can't come from a cache hit. ## Does this PR introduce _any_ user-facing changes? No. The behavior only changes when `spark.delta.sharing.jsonPredicateHints.partialFilter.enabled` is set to `true` (default `false`). With it on, queries with unsupported filters can get server-side file skipping from their remaining filters. This pull request and its description were written by Isaac. --------- Signed-off-by: Abhijit Chakankar <abhijit.chakankar@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
…or (delta-io#8173) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description `SingleCommit.getActionsIterator` returned the actions of a commit's log file as written. For an inline manifest commit of a table with an adaptive metadata tree, the `AddFile` and `RemoveFile` actions are recorded in the tree, so the iterator did not return them. Callers that process a commit's actions one at a time, without loading them all into memory, need the actions that `getActions` returns. - With `DeltaSQLConf.AMT_SINGLE_COMMIT_READ_ENABLED` on, `getActionsIterator` returns the same actions as `getActions`, in the same order: the log file's other actions, then the file actions reconstructed from the tree. With the conf off, it returns the log file as written. - The new `AMTCheckpointProviderImpl.reconstructFileActionsIterator` returns the actions of `reconstructFileActions`, in its order, and pulls them to the driver one partition at a time. The iterator opens the tree only after it has read the log file's actions. `rewind()` reads both again. - `getActions` and `getActionsIterator` share one path. A commit is an inline manifest commit when its first action is a `CommitInfo` whose `lastManifestCommit` is at the commit's own version (`CommitInfo.isAMTInlineManifestCommit`). Its tree is located from its `Checkpoint` action, and the read fails if that action is missing. - The test helper `getJsonOnlyChanges` now reads the log files directly. ## How was this patch tested? New tests in `AMTSingleCommitSuite`: - `getActionsIterator` returns the same actions after `rewind()`. - `getActions` and `getActionsIterator` return a manifest commit's actions in the same order. - `reconstructFileActionsIterator` returns the actions of `reconstructFileActions` in order, with 7 shuffle partitions. - `getActionsIterator` returns a log commit as written, and returns the log file as written when single-commit read is off. The existing `getChanges` test for an inline manifest commit also checks `getActionsIterator`. ## Does this PR introduce _any_ user-facing changes? No.
…io#8184) [AI-assisted] ## Description Adds a concurrency group to the Delta Spark workflow with `cancel-in-progress: true`. PR runs are grouped by PR number, so a new commit cancels the previous in-progress run for the same PR. Non-PR runs—pushes to `master` or `branch-*`, and `workflow_dispatch` runs—fall back to `github.run_id`, giving every run a unique concurrency group. They are therefore never cancelled, preserving a full CI result for every master commit and maintaining master regression detection. ## Testing - Parsed `.github/workflows/spark_test.yaml` successfully with PyYAML. - `git diff --check` passed. - Confirmed the final diff against `delta/master` contains only the concurrency comment and block. --------- Co-authored-by: omnigent <noreply@omnigent.ai>
…lta-io#8180) [AI-assisted] <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> Merge the Java and Scala map overload implementations of `getPartitionRow` through a shared helper. The overload-specific behavior remains unchanged, including the Java map size assertion, the Scala map missing-key handling, and lazy map access for empty partition schemas. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> - `build/sbt "sparkV2/Test/javafmtCheck"` - `build/sbt "sparkV2/testOnly org.apache.spark.sql.delta.v2.utils.PartitionUtilsTest"` (23 test cases) ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No. --------- Signed-off-by: tdas <tathagata.das1565@gmail.com>
…8181) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Re-enable `incremental: commit file gap between versions, failOnDataLoss=false succeeds` in `DeltaV2SourceSuite` by moving it from `FailingTests` to `PassingTests`. Remove the obsolete comment that describes this case as unsupported. This restores regression coverage for streaming across a gap between commit versions when `failOnDataLoss=false`. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? `git diff --check` passed. Tests were not run locally because this change only re-enables an existing test; execution is left to the Spark CI workflow. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. This change only updates test coverage. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> Signed-off-by: You Zhou <yz3883@columbia.edu>
#### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description The Delta protocol calls these tables catalog-managed, and the table feature is `catalogManaged`. Comments in the Spark connector still say "catalog-owned", "Catalog Owned" or "CatalogOwned tables". This PR changes that prose to "catalog-managed". ## How was this patch tested? Comment-only change. ## Does this PR introduce _any_ user-facing changes? No.
…s definitions (delta-io#8158) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Stacked on delta-io#8151; review the last commit. Renames the `CatalogOwnedTableFeature` and `CatalogOwnedTableUtils` objects to `CatalogManagedTableFeature` and `CatalogManagedTableUtils`. The feature name in the protocol stays `catalogManaged`. The old names stay as aliases, in the `delta` package object and in a new `coordinatedcommits` package object, so the callers keep compiling until the PRs stacked on this one move them to the new names. `UCDeltaCatalogClientImpl.scala` moves to the new name here. ## How was this patch tested? Rename only, no new tests. Apart from the aliases and the line wrapping, every changed token is an old name changed to the new one. ## Does this PR introduce _any_ user-facing changes? No.
#### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Record substantive Delta V2 snapshot loads, history requests, file-state construction, and table-manager cache lifecycle operations through the existing operation hooks. Normalize operation names under `delta.v2.` and attribute events to the table path without requiring a DeltaLog-backed provider. Cache timing uses a shared elapsed-time helper. Keep `KernelContext` focused on filesystem options and its LogStore. Remove the unused table-path constructor/factory argument and update the production and test callers; engine creation does not use that argument. Keep configuration materialization scoped to the active session with table options overlaid. Path and payload construction for table-scoped events is guarded so non-fatal telemetry failures preserve the caller's control flow and original exception. Ordinary cache hits, snapshot freshness decisions, cleanup behavior, and exceptions retain their existing behavior. This is the root of the operations stack: delta-io#8067, delta-io#8068. It is based directly on master and can be reviewed independently of the diagnostics stack (delta-io#8062, delta-io#8066). Design agreement is tracked in delta-io#8065 and remains pending. ## How was this patch tested? Both `sparkV2/Test/compile` and `spark/Test/compile` passed. Six affected suites passed all 92 tests with zero failures or skips: KernelContextSuite (7), CachedSnapshotManagerSuite (35), UCManagedTableSnapshotManagerSuite (10), DeltaV2SnapshotSuite (23), KernelSnapshotUtilsSuite (5), and ApplyV2ReadOptionsSuite (12). Style checks passed for all changed Scala files. Whitespace checks pass. CI must rerun on the updated head. ## Does this PR introduce _any_ user-facing changes? Yes. Delta V2 operation hooks gain consistent names and table attribution. Query results and the transaction-log format are unchanged. --------- Signed-off-by: Zheng Hu <openinx@gmail.com>
…ts (delta-io#8182) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> The V2 scan Java tests cast the built scan to DeltaV2Scan in about 90 places. A failed cast gives a ClassCastException that does not say which scan the builder built. Add DeltaV2TestBase.asDeltaV2Scan, which checks the type and fails with the class name of the scan, and use it in DeltaV2ScanTest, DeltaV2ScanBuilderTest, and DeltaV2BatchTest. The test assertions do not change. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> Existing tests ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No Signed-off-by: Hari Kishore Chaparala <hchapara@uci.edu> Co-authored-by: Isaac <no-reply@databricks.com>
…eltaV2MicroBatchStream (delta-io#8187) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description This PR starts to share the offset logic of the DSv1 streaming source `DeltaSource` with the DSv2 streaming source `DeltaV2MicroBatchStream`. Each PR moves one `DeltaSource` method, unchanged, into a shared trait, and deletes the `DeltaV2MicroBatchStream` copy. This PR adds the trait `DeltaSourceOffsetLogic`, which both sources mix in, and moves `checkNonAdditiveSchemaChanges` into it. `DeltaV2MicroBatchStream` now also runs the `assert(!trackingMetadataChange)` check that `DeltaSource` has. Every `DeltaV2MicroBatchStream` caller already skips the check while it tracks metadata. ## How was this patch tested? Existing tests. No test file changes. ## Does this PR introduce _any_ user-facing changes? No.
#### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other ## Description `SnapshotManagementParallelListingSuite` registers its test coordinator for `spark_catalog`. If another catalog is current, its unqualified table name creates the table elsewhere and bypasses that coordinator. Qualify the test table name and use an explicit catalog-qualified identifier when opening the Delta log so the listing reconciliation assertions exercise the intended coordinator. ## How was this patch tested? - `git diff --check` passed. - The local `spark/testOnly org.apache.spark.sql.delta.SnapshotManagementParallelListingSuite` run could not start because the sbt launcher download was unavailable. The suite will run in CI. ## Does this PR introduce _any_ user-facing changes? No. Signed-off-by: Eric Tseng <52102337+EricTsengTy@users.noreply.github.com>
delta-io#8170) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Support querying versionless CDF for delta sharing clients with delta format. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? UTs <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> --------- Co-authored-by: Omar Elhadidy <omar.elhadidy@databricks.com>
…a-io#8157) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Stacked on delta-io#8158; review the last commit. Renames the `CatalogOwnedUsageLogs` object, which holds the op types of catalog-managed tables, to `CatalogManagedUsageLogs`. The `coordinatedcommits` package object keeps the old name as an alias. The op type values, such as `delta.catalogOwned.commitToUcEtagRetry`, stay the same. ## How was this patch tested? Rename only, no new tests. Apart from the aliases and the line wrapping, every changed token is an old name changed to the new one. ## Does this PR introduce _any_ user-facing changes? No.
…builders (delta-io#8113) [AI-assisted] #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Extracts the shared row-tracking file metadata lookup, missing-value validation, and long conversion used by the V2 row ID and row commit version value setter builders. Exception types and messages remain unchanged. ## How was this patch tested? - `build/sbt "sparkV2/Compile/javafmtCheckAll"` - `build/sbt "sparkV2/testOnly org.apache.spark.sql.delta.v2.read.metadata.MetadataStructReadFunctionTest org.apache.spark.sql.delta.v2.read.metadata.MetadataStructSchemaContextTest org.apache.spark.sql.delta.v2.read.metadata.MetadataValueSetterTest org.apache.spark.sql.delta.v2.V2RowTrackingReadTest"` ## Does this PR introduce _any_ user-facing changes? No. --------- Signed-off-by: tdas <tathagata.das1565@gmail.com>
… have deletion vectors (delta-io#8190) #### Which Delta project/connector is this regarding? - [x] Spark ## Description `DeletionVectorsTableFeature.validateDropInvariants` treated an unknown DV count as zero. `numDeletionVectorsOpt` is unknown when DVs are no longer writable and the snapshot has no checksum with the count, for example after a RESTORE commit. In that case DROP FEATURE deletionVectors succeeded while files still had deletion vectors. This change checks the files themselves (`DeletionVectorUtils.isTableDVFree`) when the count is unknown. ## How was this patch tested? Tested with a scenario that drops the feature after a RESTORE leaves the DV count unknown while files still have DVs. Without this change DROP FEATURE succeeds; with it, DROP FEATURE fails with `DELTA_FEATURE_DROP_CONFLICT_REVALIDATION_FAIL`. ## Does this PR introduce _any_ user-facing changes? No. Co-authored-by: Isaac <no-reply@databricks.com>
delta-io#8194) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description A clustered table can be clustered by a field inside a struct, for example `a.b`. DROP COLUMNS rejects a drop of a clustering column, but it compares only whole names. `DROP COLUMNS (a)` drops the struct `a`, and `a.b` with it. The check misses that, because `"a"` is not `"a.b"`. The table is then clustered by a column that is gone, and later writes, DESCRIBE DETAIL and OPTIMIZE fail with `DELTA_CANNOT_RESOLVE_CLUSTERING_COLUMN`. The check now also rejects a drop of a struct that holds a clustering column: it matches when the name parts of a dropped column are a prefix of the name parts of a clustering column. ## How was this patch tested? New test in `ClusteredTableDDLSuite`: on a table clustered by `a.b`, `DROP COLUMNS (a)`, `(A)` and `(a.b)` fail with `DELTA_UNSUPPORTED_DROP_CLUSTERING_COLUMN`, and `DROP COLUMNS (a.c)` succeeds and keeps `a.b` as the clustering column. ## Does this PR introduce _any_ user-facing changes? Yes. `DROP COLUMNS` of a struct that holds a clustering column now fails with `DELTA_UNSUPPORTED_DROP_CLUSTERING_COLUMN`. Before, it succeeded and left the table clustered by a missing column.
) <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Make `DeltaDataFrameWriterV2Suite` easier to reuse with other table providers while preserving its existing Delta expectations: - Extract partition assertions into overridable `assertPartitionedById` and `assertUnpartitioned` helpers. - Use the existing `createTableSQL` and `writeFormat` helpers in the literal-string overwrite tests instead of hardcoded provider names. - Remove an unused import. <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> ## How was this patch tested? `git apply --check` and `git diff --check` passed. The test suite has not been run in this checkout. <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ## Does this PR introduce _any_ user-facing changes? No. This only refactors test scaffolding; the default assertions and provider remain unchanged. <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> Signed-off-by: Tejas Maire <36109575+tmaire111@users.noreply.github.com>
…elta-io#8121) [AI-assisted] #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description `DeltaV2ScanUtils.newScanBuilder` was a pass-through to `DeltaV2ScanBuilder.create`, which only derived the snapshot manager from the table manager (`tableManager.snapshotManager(catalogTable)`) and called the `DeltaV2ScanBuilder` constructor. Two factory layers did one job. This PR removes the extra layer: - `DeltaV2ScanUtils.newScanBuilder` now constructs the `DeltaV2ScanBuilder` directly, with the same arguments `create` used. It stays the single entry point because `DeltaV2ScanBuilder` is `private[read]` and its production caller, `DeltaV2Table`, lives in another package. It also returns only the `ScanBuilder` interface. - `DeltaV2ScanBuilder.create` is deleted. - `DeltaV2ScanBuilderTest` now builds its scan builder through `DeltaV2ScanUtils.newScanBuilder`, the same path `DeltaV2Table` uses. This is a pure refactor with no behavior change. ## How was this patch tested? Existing tests: - `build/sbt "sparkV2/Test/javafmtCheck" "sparkV2/javafmtCheckAll"` passes. - `build/sbt "sparkV2/testOnly org.apache.spark.sql.delta.v2.read.DeltaV2ScanBuilderTest org.apache.spark.sql.delta.v2.read.DeltaV2ScanTest"` passes all 120 test cases: 35 in `DeltaV2ScanBuilderTest` and 85 in `DeltaV2ScanTest`. ## Does this PR introduce _any_ user-facing changes? No. This pull request and its description were written by Isaac. Signed-off-by: tdas <tathagata.das1565@gmail.com> Co-authored-by: Isaac <no-reply@databricks.com> Co-authored-by: omnigent <noreply@omnigent.ai>
…lta-io#8118) [AI-assisted] #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Clarify that the limit-aware scan receives partition-only filters because `build()` only sets `effectiveLimit` when there are no residual data filters. ## How was this patch tested? - `sparkV2/Compile/compileIncremental` - `sparkV2/Compile/javafmtCheckAll` - `sparkV2/Compile/scalafmtCheckAll` (blocked by pre-existing formatting failures) ## Does this PR introduce _any_ user-facing changes? No. Signed-off-by: tdas <tathagata.das1565@gmail.com>
…ts (delta-io#8202) ## Summary - Run coordinator retry and `commitLarge` failure tests against catalog-managed Delta tables. - Preserve the retry limits, exception checks, and failure telemetry assertions. - Remove a legacy property-based commit-directory test and mark four coordinator cases as expected failures in the V2 interop suite. ## Testing - `git diff --check` - Focused sbt test could not start because the sbt launcher download failed in the local checkout. Public CI will run the suite. Signed-off-by: Eric Tseng <52102337+EricTsengTy@users.noreply.github.com>
…Coordinator* (delta-io#8153) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Stacked on delta-io#8157; review the last commit. Renames `CatalogOwnedCommitCoordinatorBuilder` and `CatalogOwnedCommitCoordinatorProvider` to `CatalogManagedCommitCoordinatorBuilder` and `CatalogManagedCommitCoordinatorProvider`. The `coordinatedcommits` package object keeps both old names as aliases. ## How was this patch tested? Rename only, no new tests. Apart from the aliases and the line wrapping, every changed token is an old name changed to the new one. ## Does this PR introduce _any_ user-facing changes? No.
…nabled (delta-io#8205) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Adds a required `isPreviewCompatible` parameter to `AMTUtils.amtSupported` and `AMTUtils.amtEnabled`, so every caller states whether it handles `adaptiveMetadata-preview` tables. With `false`, preview tables don't count as AMT tables; `adaptiveMetadata-dev` tables behave as before. New code paths, such as `tightStats`, pass `false` so they don't run on preview tables, which older clients still read and write. Every existing caller passes `true`, so this PR doesn't change behavior. The single-argument `amtEnabled(snapshot)` overload is removed; callers use `amtEnabled(snapshot, isPreviewCompatible)`. ## How was this patch tested? `AMTUtilsSuite` checks every feature and AMT property case with the flag set to `true` and to `false`. `AdaptiveMetadataTableFeatureSuite` checks the snapshot form on real tables: with `false`, only `adaptiveMetadata-dev` tables count as AMT tables. ## Does this PR introduce _any_ user-facing changes? No.
…ta-io#8206) #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (Delta Sharing) ## Description Removes the internal conf `spark.sql.delta.sharing.enableAutoResolveForCdf` (`DeltaSQLConf.DELTA_SHARING_ENABLE_AUTO_RESOLVE_FOR_CDF`). A Delta Sharing batch CDF read without an explicit `responseFormat` now always auto-resolves its response format. - `DeltaSharingDataSource.createRelation`: every batch read without a `responseFormat`, snapshot or CDF, goes to `autoResolveBaseRelation`. That method calls getMetadata, pinned to the end of the CDF range (or the latest version when the range has no end), and takes the parquet or delta path that the server answers with. Before this change, a CDF read with the conf off (its default) fell back to the legacy parquet path. The `shouldAutoResolveCDF` helper is removed. Reads with an explicit `responseFormat` are unchanged. - `DeltaSQLConf`: the conf is removed. - `DeltaSharingDataSourceDeltaSuite`: the two tests parameterized on the conf are replaced by always-on versions. One reads delta-format CDF across version and timestamp bounds. The other reads a parquet-format share and checks that it resolves to the parquet path (the parquet CDF reader's `LongType` `_commit_timestamp`). The conf was added so the CDF auto-resolve path could be rolled out gradually. That rollout is done, so the conf is no longer needed. ## How was this patch tested? Updated the tests in `DeltaSharingDataSourceDeltaSuite` as described above. - `build/sbt "spark/scalastyle" "sharing/scalastyle" "sharing/Test/scalastyle" "sharing/testOnly io.delta.sharing.spark.*"` Result: scalastyle reports 0 errors and 0 warnings. `sharing/testOnly`: 130 passed, 2 failed. The same 2 tests fail with this command on unmodified master (3b1bb78): `DeltaSharingDataSource able to auto resolve responseFormat` fails only when run after `DeltaFormatSharingSourceSuite` in the same JVM (the global `requestedFormat` map collects `shared_parquet_table*` entries from that suite) and passes when its suite runs alone. `versionless CDF refresh preserves timestamp bounds and isolates queries` (`DeltaSharingCDFUtilsSuite`) hits a 20s `eventually` timeout. ## Does this PR introduce _any_ user-facing changes? Yes. A Delta Sharing batch CDF read without a `responseFormat` option (for example `spark.read.format("deltaSharing").option("readChangeFeed", "true").load(path)` or `table_changes`) now asks the server for the table's format and reads through the delta-format path when the server answers delta. Before, by default it always used the parquet path. The removed conf was internal. This pull request and its description were written by Isaac. Signed-off-by: lin.zhou <lin.zhou@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
…nseFormat (delta-io#8207) #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [x] Other (Delta Sharing) ## Description Removes the internal conf `spark.sql.delta.sharing.cdfStreamingAutoResolveResponseFormat` (`DeltaSQLConf.DELTA_SHARING_CDF_STREAMING_AUTO_RESOLVE_RESPONSE_FORMAT`). Delta Sharing streaming CDF (`readChangeFeed=true`) now always auto-resolves its response format. - `DeltaSharingDataSource.autoResolveStreamingSource`: a CDF stream always uses getMetadata. When `responseFormat` is not set, it takes the format the server answers with, and an explicit `responseFormat=delta` goes to the delta-format source. Non-CDF streaming is still gated by `spark.sql.delta.sharing.streamingAutoResolveResponseFormat`. - `DeltaFormatSharingSource.isAutoResolveEnabled`: always true for CDF streams, so the delta-format source accepts a legacy `DeltaSharingSourceOffset` checkpoint left by an earlier parquet CDF stream and picks the matching file-id hash. - `DeltaSQLConf`: the conf is removed. - `DeltaFormatSharingSourceSuite`: the `forceToDeltaSourceOffset` flag-routing grid now covers CDF auto-resolve with the non-CDF conf both on and off, and non-CDF with the non-CDF conf on and off. The conf was added so CDF streaming auto-resolve could be rolled out separately from non-CDF streaming. That rollout is done, so the conf is no longer needed. ## How was this patch tested? Updated the routing-grid test in `DeltaFormatSharingSourceSuite` as described above. - `build/sbt "spark/scalastyle" "sharing/scalastyle" "sharing/Test/scalastyle" "sharing/testOnly io.delta.sharing.spark.*"` Result: scalastyle reports 0 errors and 0 warnings. `sharing/testOnly`: 130 passed, 2 failed. The same 2 tests fail with this command on unmodified master (3b1bb78): `DeltaSharingDataSource able to auto resolve responseFormat` fails only when run after `DeltaFormatSharingSourceSuite` in the same JVM (the global `requestedFormat` map collects `shared_parquet_table*` entries from that suite) and passes when its suite runs alone. `versionless CDF refresh preserves timestamp bounds and isolates queries` (`DeltaSharingCDFUtilsSuite`) hits a 20s `eventually` timeout. ## Does this PR introduce _any_ user-facing changes? Yes. A Delta Sharing streaming CDF read now always asks the server for the table's format. If `responseFormat` is not set and the server answers delta, the stream uses the delta-format source even when `spark.sql.delta.sharing.streamingAutoResolveResponseFormat` is off. Before, by default it used the parquet source. A CDF stream that restarts with `responseFormat=delta` from a legacy parquet CDF checkpoint now resumes from that checkpoint instead of failing to parse it. The removed conf was internal. This pull request and its description were written by Isaac. Signed-off-by: lin.zhou <lin.zhou@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
…#8208) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description The V4 spec lets a leaf manifest hold both live files and tombstones, and other engines write such leaves, for example Iceberg after a delete and insert. The incremental AMT writer rejected such a leaf when it carried the leaf forward, so every later write on the table failed with: ``` [INTERNAL_ERROR] ManifestInfo representing a leaf's initial state must not contain both live files and tombstones. ``` Readers already skip a carried leaf's tombstones by entry status, so this writer check was the only thing in the way. This PR: - Removes the check and its alert. - Records a non-fatal usage log instead, once per incremental write that carries mixed leaves, with how many it carried. - Picks a carried leaf's tracking from its live files only. Before, any leaf with a tombstone went straight to DELETED, which would have dropped a mixed leaf's live files. - Assumes a manifest deletion vector masks live files only, as the path that adds new masking already does. A leaf becomes DELETED once its deletion vector covers all of its live files. A deletion vector that covers more entries than the leaf has live files fails the write with the `amt.commit.mdvExceedsLiveFiles` alert instead of guessing which entry is still live. The V4 spec wants a leaf MODIFIED in the commit that masks its last live file and DELETED in the next commit. The writer still marks it DELETED right away, as before. ## How was this patch tested? New tests in `AMTIncrementalWriteInvariantSuite`: - Unit tests for `carryForwardOneLeaf` with a mixed leaf (EXISTING, MODIFIED and DELETED outcomes), a mixed leaf an earlier commit fully or partly masked, a manifest deletion vector that masks more entries than the live files, and live-only and tombstone-only leaves for comparison. - An end-to-end test that swaps one leaf of a real AMT table for a copy with an extra DELETED entry, then runs INSERTs and DELETEs on top. At each step it checks the leaf's tracking, the change feed (the tombstone never shows up as a remove), and the table's rows. It also checks that the first write records the mixed-leaf usage log once. Without the fix, the first write fails with the error above. ## Does this PR introduce _any_ user-facing changes? No. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
…#8172) ## Description When a snapshot discovers an AMT checkpoint from storage, the columnar provider can keep the root's rows in memory and reuse them. A warm commit does not go through discovery: it builds the next provider from `AMTWriteResult`, which only carried the leaf pointers. So the next incremental AMT write had to read the root Parquet file again. This PR carries the complete `AMTRootInfo`, leaf pointers plus any cached root rows, through `AMTWriteResult`. It keeps the rows the writer produced when the root is below the root-cache threshold: - Incremental writes and multi-leaf full writes keep the rows built on the driver to write the root. - When a full write produces a single leaf that is promoted to the root, no rows are kept, and the next commit reads that root from storage. The writer and discovery use the same check, `AMTUtils.shouldCacheRootRows`: columnar state reconstruction is enabled and the root has fewer entries than `amt.rootCache.maxNumEntries`. The cached rows are copied into unsafe rows so they don't depend on the writer's buffers. Turning the written rows into a local relation now applies `ConvertToLocalRelation` until the plan stops changing (`AMTUtils.convertToLocalRelationFully`). When the rule visits the plan top-down, one application folds only the projection directly above the local relation, which made the writer silently skip caching. The writer now asserts that the rows fold to a local relation. Decoding leaf pointers from a cached root uses the same helper. No extra Spark scan is added. Large roots, and the case where root caching is off (`amt.rootCache.maxNumEntries` = 0), keep reading the root from storage as before. This PR also raises the default of `amt.rootCache.maxNumEntries` from 12000 to 50000. ## How was this patch tested? New tests in `AMTSnapshotDiscoverySuite`: - After a warm commit, the columnar provider reuses the root rows the writer produced. - The writer does not keep root rows when the root is at or above the threshold, or when caching is off. - Discovering a cached root decodes its leaf pointers without launching another Spark job. `LastCheckpointInfoSuite` is updated for the new `AMTWriteResult` shape. ## Does this PR introduce _any_ user-facing changes? No. Co-authored-by: Jinhua Song <jinhua.song@databricks.com>
#### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Stacked on delta-io#8153; review the last commit. Moves callers of `CatalogOwnedTableFeature` to the new name `CatalogManagedTableFeature`. Each change is the name only. ## How was this patch tested? Rename only, no new tests. Apart from the aliases and the line wrapping, every changed token is an old name changed to the new one. ## Does this PR introduce _any_ user-facing changes? No.
…ta-io#8203) #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [x] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description When a Flink job is resubmitted from a checkpoint with a new job ID, the committer uses the new ID to replay work created by the old job. Delta treats the replay as a new transaction. For example, if OPTIMIZE has replaced an old file, replay can add that file back to the active table. If VACUUM has deleted it, the table can then reference a missing file. Use the job ID saved in each committable to identify its transaction. Group work by that saved job ID before sorting it by checkpoint number, so recovered work and new work stay separate even when their checkpoint numbers match. The checkpoint serialization format stays unchanged. The committer builder no longer needs a job ID; `withJobId()` remains as a deprecated no-op for compatibility. Upgrade caveat: if an older connector first committed checkpoint work under a replacement job ID, and there is no transaction record for its original job ID, restoring that checkpoint after upgrading can replay it once more. This change does not repair transaction history written by older versions. ## How was this patch tested? local and CI tests. ## Does this PR introduce _any_ user-facing changes? Restores the intended behavior when resubmitting a job from a checkpoint with a new job ID: already committed work is skipped, files removed by later transactions stay removed, and new work still commits. --------- Signed-off-by: Timothy Wang <31230953+timothyw553@users.noreply.github.com>
…8215) [AI-assisted] <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md 2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'. 3. Be sure to keep the PR description updated to reflect all changes. 4. Please write your PR title to summarize what this PR proposes. 5. If possible, provide a concise example to reproduce the issue for a faster review. 6. If applicable, include the corresponding issue number in the PR title and link it in the body. --> #### Which Delta project/connector is this regarding? <!-- Please add the component selected below to the beginning of the pull request title For example: [Spark] Title of my pull request --> - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description <!-- - Describe what this PR changes. - Describe why we need the change. If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge. --> This test-only follow-up to delta-io#8105 exercises the production metadata schema conversion path with the converter's former test inputs. It also adds a smoke test for UTF8_BINARY string schemas, including array and nested-struct shapes. ## How was this patch tested? <!-- If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible. If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future). If the changes were not tested, please explain why. --> ```shell build/sbt "sparkV2/Test/javafmtCheck" "sparkV2/javafmtCheck" build/sbt "sparkV2/testOnly org.apache.spark.sql.delta.v2.utils.SchemaUtilsTest" ``` `SchemaUtilsTest` passes all 44 test cases. ## Does this PR introduce _any_ user-facing changes? <!-- If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible. If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master. If no, write 'No'. --> No. --------- Signed-off-by: tdas <tathagata.das1565@gmail.com> Co-authored-by: omnigent <noreply@omnigent.ai>
#### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [x] Kernel - [ ] Other (fill in here) ## Description Java Kernel searches for checkpoints in fixed 1,000-version batches. On tables with frequent multipart checkpoints, a batch can contain many objects and make snapshot loading expensive. This PR adds `SnapshotBuilder.withCheckpointSearchBatchSize(int)` so callers can choose a positive batch width per snapshot. Kernel uses that width for both the initial search and stale-checkpoint fallback, and continues into older batches until it finds a complete checkpoint. The default remains 1,000 versions. The internal Checkpointer and full SnapshotManager calls require the width explicitly. Checkpoint search also closes each listing iterator and stops after checking the zero-start batch once. ## How was this patch tested? Added tests for a width of 5 across older batches and stale-checkpoint fallback, nonpositive width rejection, missing and incomplete checkpoints at the zero-start batch, and listing-iterator closure across batches and on return. From the repository root with JDK 17: ```bash sbt --java-home /usr/lib/jvm/java-17-openjdk-amd64 -batch -no-colors 'kernelApi/testOnly io.delta.kernel.*' ``` Passed: 2,985 tests, 0 failures, 0 errors. ## Does this PR introduce _any_ user-facing changes? Yes. `SnapshotBuilder` gains `withCheckpointSearchBatchSize(int)`. Previously, checkpoint discovery always searched 1,000 versions per batch. Callers can now set a positive width for a snapshot; callers that do not set it retain the 1,000-version default. --------- Signed-off-by: Thinh Bui <26578143+thinh2@users.noreply.github.com>
…elta-io#8218) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Match partition columns by their already-resolved names in `DeltaV2ScanBuilder.pruneColumns`, instead of lowercasing both sides. This preserves a metadata field named `__metadata` beside a partition column named `__Metadata`, while still excluding the partition column from the data-read schema. CDC column filtering remains case-insensitive. ## How was this patch tested? Added a regression test using a Delta table with a stored `_metadata` column and a `__Metadata` partition column. The test passes a resolved `__metadata` metadata field directly to the scan builder and verifies that pruning preserves it under both case-sensitivity settings. Local test execution is deferred to CI. `git diff --check` passed. ## Does this PR introduce _any_ user-facing changes? Yes. V2 column pruning preserves resolved metadata fields whose names differ from partition-column names only by case, instead of discarding them. Signed-off-by: You Zhou <yz3883@columbia.edu>
…io#8219) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other ## Description V2 streaming reads commit JSON through LogStore. Validate each protocol action with the shared `Protocol.protocolRead` check before returning the commit's files or collecting metadata and protocol actions. This rejects an unsupported historical protocol even when the latest snapshot has a supported protocol. Pass the stream's Spark session and read snapshot to the action collectors. Add coverage for unsupported reader versions and reader features, supported writer-only features, lazy validation, and reader cleanup. Enable the existing historical-protocol streaming regression tests. ## How was this patch tested? - Four focused suites passed with Java 17 and Spark 4.2: **325 tests passed, 0 failures** (`StreamingHelperTest`, `DeltaV2MicroBatchStreamTest`, `MetadataEvolutionHandlerTest`, and `DeltaV2TableManagerImplSuite`). - `git diff --check` - Java formatting checks with Java 17 and Spark 4.2: ```sh ./build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 -DsparkVersion=4.2 \ 'sparkV2/javafmtCheckAll' ``` ## Does this PR introduce _any_ user-facing changes? On master, V2 streams now reject unsupported historical reader protocols with the standard Delta protocol-read errors before emitting files from that commit. Unknown writer-only features remain readable. No public API or table format changes. Signed-off-by: Hao Sun <86338940+HaoSun1993@users.noreply.github.com>
…elta-io#8214) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description The stats JSON of AMT entries is now built from a single stats struct, assembled out of plain field accesses into the typed `content_stats` column, instead of rebuilding the nested shape path by path. Statistics are matched to Delta's stats schema by physical path, a matched statistic of a widened type is cast, and statistics the manifests do not persist read as null. The JSON itself is unchanged, including leaving out min/max groups that have no persisted values rather than writing `{}`. ## How was this patch tested? Existing AMT stats and state reconstruction suites, plus a new `AMTContentStatsSuite` test for a table where no column has persisted bounds. ## Does this PR introduce _any_ user-facing changes? No Co-authored-by: Murali Ramanujam <murali.ramanujam@databricks.com>
…AMT stats (delta-io#8223) #### Which Delta project/connector is this regarding? - [x] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description Add an `AMTFieldSpecSuite` test that drops a column and re-adds a column with the same name, which gets a new column mapping id. In each AMT checkpoint scenario, it checks that: - the tree written before the drop holds min/max/null-count stats for the dropped column, - the files written before the drop carry no stats for the re-added column, with Parquet field-id reads on and off, and the new file keeps its stats, - an `IS NULL` filter on the re-added column returns every row of the old files. ## How was this patch tested? Test-only change. ## Does this PR introduce _any_ user-facing changes? No.
#### Which Delta project/connector is this regarding? - [x] Spark ## Description Sharing checkpointed setup across compatible AMT back-reference checks avoids repeated fixture construction while keeping clone targets isolated. | Pre-export source-suite workload and timed scope | Before | After | Benefit | | --- | ---: | ---: | ---: | | Complete five-shard `AMTBackReferenceSuite`, mean command wall time | 463.26 s | 430.72 s | 7.0% lower | Two alternating pre-export runs per source revision compared `a3ac97a194a85815a3ec9572939e882e4f053aa0` with `19a2cde8bfd72b4087188595489f482eed6a2455`, including startup, fixtures and teardown. Later source cleanups and this public head were not timed. CPU and CI savings are unmeasured. ## How was this patch tested? The 75-case pre-export source suite retains all original obligations across three checkpoint modes, with deliberate back-reference faults and alternate-order checks. Rejected transactions preserve table version and live references. The generated patch was applied to public master. The cleanup preserves the shared helper assertions and was checked in all three Runtime checkpoint modes. The transformed public suite has not been run locally. All 53 public CI checks pass on `a9d933322ec58ebc3d3a39fa8dc397df711af3ae`. ## Does this PR introduce any user-facing changes? No. <details> <summary>Agent attribution</summary> Agent: Costly Tests - Orchestrator | Session: 01a1e505-448e-7d81-b315-7ef768b0399c Agent: Costly Tests - SQL | Session: 01a1e523-4d27-73b1-b47b-a8e576721388 </details> --------- Signed-off-by: Cyrus Najmabadi <cyrus.najmabadi@databricks.com> Signed-off-by: cyrus.najmabadi <cyrus.najmabadi@databricks.com> Co-authored-by: Cyrus Najmabadi <cyrus.najmabadi@databricks.com>
#### Which Delta project/connector is this regarding? - [x] Spark ## Description Sharing checkpoints across compatible AMT snapshot checks avoids repeated fixture construction while keeping mutable histories isolated. | Pre-export source-suite workload and timed scope | Before | After | Benefit | | --- | ---: | ---: | ---: | | Complete five-shard `AMTSnapshotSuite`, mean first test-action start through last test-action finish | 371.740 s | 352.347 s | 5.22% lower | Two alternating pre-export runs per source revision compared `0af7c6e203e660d0073f25e2b0c261ecfa0e527d` with `408de8dba1118392ce5e6018424760af41cb7fa3`, including startup, fixtures and teardown. Later source cleanups and this public head were not timed. CPU and CI savings are unmeasured. ## How was this patch tested? The 87-case pre-export source suite retains all 99 original obligations across three checkpoint modes, with alternate-order and deliberate DELETE/manifest-deletion-vector faults. The final generated patch was applied to public master, and the resulting file matches the reviewed output blob. The transformed public suite has not been run locally. Public CI passes on `1849c03b526045aee1b42e7dd43c467ca159fcf3`. ## Does this PR introduce any user-facing changes? No. <details> <summary>Agent attribution</summary> Agent: Costly Tests - Orchestrator | Session: 01a1e505-448e-7d81-b315-7ef768b0399c </details> Signed-off-by: Cyrus Najmabadi <cyrus.najmabadi@databricks.com> Co-authored-by: Cyrus Najmabadi <cyrus.najmabadi@databricks.com>
Adds a metadata-only command that removes the partitioning of a Delta table without rewriting data files: ALTER TABLE t REPLACE PARTITIONED BY WITH CLUSTER BY ALTER TABLE t REPLACE PARTITIONED BY WITH CLUSTER BY (c1, c2) ALTER TABLE t REPLACE PARTITIONED BY WITH CLUSTER BY NONE Without columns, the partition columns become the clustering columns; with NONE the table becomes a plain unpartitioned table. The command verifies that every data file physically stores the partition columns (written when delta.writePartitionColumnsToParquet is enabled), then re-adds all files in a single commitLarge commit with empty partitionValues and dataChange=false. The stats of each file are extended with min/max/nullCount for the former partition columns, and delta.dataSkippingStatsColumns is set when they would otherwise lose stats, so data skipping on them keeps working. The existing stats are kept exactly. Tables with IcebergCompat, UniForm or adaptive metadata enabled are rejected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Eunjin Song <sezruby@gmail.com>
sezruby
force-pushed
the
replace-partitioned-by-cluster-by
branch
from
November 18, 2026 17:07
70a746e to
685eec2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
Adds a metadata-only way to remove the partitioning of a Delta table, optionally replacing it with liquid clustering:
Today the only way to drop partitioning is to rewrite the whole table (CTAS / REPLACE TABLE). For large tables that are over-partitioned (e.g. partitioned by a high-cardinality ingestion id), that is very expensive. However, since
delta.writePartitionColumnsToParquet(delta-io#6525, default on) /materializePartitionColumns(delta-io#5636), data files already contain the partition column values physically, so the partitioning can be dropped by only rewriting the log.The syntax pairs
REPLACE PARTITIONED BYwith the existingCLUSTER BY (...) | CLUSTER BY NONEclause rather than introducing a new standalone command, so it composes with the existing clustering DDL.Semantics
DELTA_REPLACE_PARTITIONED_BY_PARTITION_COLUMNS_NOT_MATERIALIZED, reporting the count and example paths. (Internal confspark.databricks.delta.alterTable.replacePartitionedBy.verifyMaterializedPartitionColumnscan disable this check.)commitLarge, like RESTORE/CLONE, so any concurrent commit aborts it):metaDatawithpartitionColumns = [](+ clustering table properties when columns are given);partitionValues = {},dataChange = false, and stats extended withminValues/maxValues/nullCountfor the former partition columns (string/integral/decimal/date; null count only for other types), so data skipping on former partition columns continues to work. Partition values are parsed the same way they are read as partition values. An empty string (which the protocol defines asnull, but which Delta reads as""for string columns) or a value that cannot be cast to the column type gets no stats, since the value stored in the file is not known;delta.dataSkippingNumIndexedColscolumns, ordelta.dataSkippingStatsColumnsis set),delta.dataSkippingStatsColumnsis set to the columns that had stats before followed by the former partition columns. Partition columns cannot be listed in that property while the table is partitioned. Without this, a wide table would silently lose data skipping on the former partition columns, andCLUSTER BY (<former partition column>)would fail the stats-schema check;CLUSTER BY NONEdoes not change the protocol.CLUSTER BY (cols)enables the clustering table feature, same asALTER TABLE ... CLUSTER BY.baseRowId,defaultRowCommitVersion) and column mapping are preserved. Time travel to pre-conversion versions keeps working.Follow-ups / limitations
nullCountstats only.DELTA_SCHEMA_CHANGED_WITH_VERSIONon every restart and must be restarted with a new checkpoint. A stream withschemaTrackingLocationfails once withDELTA_STREAMING_METADATA_EVOLUTIONand continues after a restart. Relaxing the check is not safe: a stream catching up from an older version also reads files that were removed before the conversion, and those were never checked for the partition columns.How was this patch tested?
New
ReplacePartitionedByWithClusterBySuite(parser, no-rewrite NONE conversion, stats + data skipping, writes/UPDATE/DELETE/MERGE/OPTIMIZE after conversion, conversion to clustered, catalog tables with null partitions, column mapping, DV + row tracking, error cases, empty-string partition values, wide tables (delta.dataSkippingStatsColumnsset; stats kept for a partition column before and after the 32nd column, for old and new files), an existingdelta.dataSkippingStatsColumns, streaming queries with and without schema tracking). Also ranDeltaSqlParserSuite,DeltaErrorsSuite,DeltaThrowableSuiteand scalastyle.Does this PR introduce any user-facing changes?
Yes, new SQL syntax
ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY (...) | NONE.Follow-up (stacked): #18 rewrites data files that do not store the partition columns instead of failing.