Skip to content

Exclude superseded producer epochs from maintenance readiness - #669

Merged
zzylol merged 1 commit into
mainfrom
fix/maintenance-active-epochs
Sep 11, 2026
Merged

zzylol merged 1 commit into
mainfrom
fix/maintenance-active-epochs

Conversation

@zzylol

@zzylol zzylol commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Why

After a producer partition restarts, the coordinator could still use its old barrier to report readiness and return both old and new epoch inputs in the same batch.

What

Select one active epoch per partition from both staged inputs and watermarks. Ready batches contain only that epoch, and newly arriving superseded-epoch inputs are rejected before checkpoint mutation.

How

Readiness and barrier completeness use the same epoch map. Observing a new staged epoch invalidates the prior epoch's readiness immediately; the new epoch must supply its own inputs and barriers. Existing identical retries remain idempotent. Historical metadata is retained.

Before this PR

Stage epoch 1 inputs and barriers, then stage epoch 2 inputs without barriers: the coordinator incorrectly returns ready, including stale contributions.

After this PR

That batch waits for both epoch 2 barriers and contains exactly the two epoch 2 inputs. Restart preserves the decision. A new epoch 1 input cannot mutate staged state after epoch 2 is observed.

Verification

Limits

This fixes the existing coordinator's readiness contract; it does not wire continuous ingestion or claim automatic maintenance completion. Superseded metadata is retained for audit/retry, not garbage-collected. Recovery of uncommitted older-epoch work needs an explicit valid completion/replay proof; this change does not infer one.

@zzylol
zzylol merged commit 8124999 into main Sep 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant