Skip to content

[Spark] Selective file selection for REORG TABLE APPLY (PURGE) - #22

Draft
sezruby wants to merge 1 commit into
masterfrom
reorg-purge-selective
Draft

sezruby wants to merge 1 commit into
masterfrom
reorg-purge-selective

Conversation

@sezruby

@sezruby sezruby commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

Resolves #21.

REORG TABLE ... APPLY (PURGE) rewrites every file that has a deletion vector. This PR adds three opt-in session confs to narrow that selection. Defaults keep today's behavior.

Conf (spark.databricks.delta. prefix) Default Meaning
reorg.purge.minDeletedRowsRatio 0.0 Only rewrite a DV file whose deleted/physical rows ratio is ≥ R. Files without row-count stats are always eligible.
reorg.purge.minStableDuration 0 Skip files that a data-changing commit gave a DV within this duration.
reorg.purge.maxDeletionVectorAge unset Deadline: always rewrite a DV file that has had no new DV within this duration, overriding the two filters.
purge(file) = (ratio >= R AND no DV within S) OR (no DV within A)

Implementation:

  • DeltaPurgeOperation takes an optional DeltaPurgeFileSelection and applies it to files with DVs. Files that only need dropped-column cleanup are always kept.
  • To find recent DVs, getActiveCommitAtTime(now - W) resolves the window start (using in-commit timestamps when enabled). getChanges then scans from that version to the snapshot version and keeps AddFiles with dataChange = true and a DV. Paths are canonicalized, and only candidate paths are tracked, so driver memory is bounded by the candidates rather than by history.
  • If the log no longer covers a window, the check stays conservative: files count as recent for the stability check and as past the deadline for the age bound.
  • DeltaReorgTableCommand gets applyPurgeFileSelection (default false). Only the SQL analysis path sets it, so internal purges still rewrite every DV file. For example, DeletionVectorsPreDowngradeCommand, which runs when dropping the deletionVectors feature.

How was this patch tested?

New DeltaReorgPurgeFileSelectionSuite uses a partitioned table, one file per partition, with commit timestamps backdated. It covers:

  • default confs: all DV files are purged;
  • the ratio filter, including inclusiveness at the exact threshold (10/100 at R = 0.1);
  • the stability filter;
  • ratio and stability combined;
  • the deadline overriding both filters;
  • the deadline alone being a no-op;
  • a window older than the table's history;
  • a DV commit recorded with an unqualified absolute path. This test fails without path canonicalization.
  • DROP FEATURE deletionVectors still purging every DV with restrictive confs set.

DeltaReorgSuite and DeltaReorgPurgeFileSelectionSuite pass (18/18), and scalastyle passes.

Does this PR introduce any user-facing changes?

Yes: three new optional SQL confs. Behavior is unchanged unless they are set.

REORG TABLE ... APPLY (PURGE) rewrites every file with a deletion vector.
Add three opt-in session confs to narrow it:

- reorg.purge.minDeletedRowsRatio: only rewrite files whose deleted/physical
  rows ratio is at least this value.
- reorg.purge.minStableDuration: skip files that a data-changing commit gave
  a deletion vector within this duration.
- reorg.purge.maxDeletionVectorAge: always rewrite files that have had no new
  deletion vector within this duration, bounding purge latency.

Defaults keep the current behavior. Internal purges (e.g. dropping the
deletionVectors feature) ignore the confs and still purge every file.

Closes #21

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request][Spark] Selective file selection for REORG TABLE APPLY (PURGE)

1 participant