Repository navigation
Conversation
REORG TABLE ... APPLY (PURGE) rewrites every file with a deletion vector. Add three opt-in session confs to narrow it: - reorg.purge.minDeletedRowsRatio: only rewrite files whose deleted/physical rows ratio is at least this value. - reorg.purge.minStableDuration: skip files that a data-changing commit gave a deletion vector within this duration. - reorg.purge.maxDeletionVectorAge: always rewrite files that have had no new deletion vector within this duration, bounding purge latency. Defaults keep the current behavior. Internal purges (e.g. dropping the deletionVectors feature) ignore the confs and still purge every file. Closes #21 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
Resolves #21.
REORG TABLE ... APPLY (PURGE)rewrites every file that has a deletion vector. This PR adds three opt-in session confs to narrow that selection. Defaults keep today's behavior.spark.databricks.delta.prefix)reorg.purge.minDeletedRowsRatio0.0reorg.purge.minStableDuration0reorg.purge.maxDeletionVectorAgeImplementation:
DeltaPurgeOperationtakes an optionalDeltaPurgeFileSelectionand applies it to files with DVs. Files that only need dropped-column cleanup are always kept.getActiveCommitAtTime(now - W)resolves the window start (using in-commit timestamps when enabled).getChangesthen scans from that version to the snapshot version and keepsAddFiles withdataChange = trueand a DV. Paths are canonicalized, and only candidate paths are tracked, so driver memory is bounded by the candidates rather than by history.DeltaReorgTableCommandgetsapplyPurgeFileSelection(defaultfalse). Only the SQL analysis path sets it, so internal purges still rewrite every DV file. For example,DeletionVectorsPreDowngradeCommand, which runs when dropping thedeletionVectorsfeature.How was this patch tested?
New
DeltaReorgPurgeFileSelectionSuiteuses a partitioned table, one file per partition, with commit timestamps backdated. It covers:DROP FEATURE deletionVectorsstill purging every DV with restrictive confs set.DeltaReorgSuiteandDeltaReorgPurgeFileSelectionSuitepass (18/18), and scalastyle passes.Does this PR introduce any user-facing changes?
Yes: three new optional SQL confs. Behavior is unchanged unless they are set.