fix: bound maintenance replay state by materialization lifecycle - #607
Merged
Merged
Conversation
zzylol
changed the base branch from
fix/maintenance-publication-retry
to
main
September 11, 2026 00:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Maintenance replay state retained every serialized input and derived payload after publication, independently of SummaryStore retention. Its query-local node IDs could also collide across installed DAGs.
Before this PR, completed panes accumulated for the process lifetime and unrelated materializations could share a publication key. After this PR, receipts use target summary-definition identity and SHA-256 input lineage, accepted payloads are released, and receipt retention follows the installed materialization's event-time retention span. Expired replay is rejected instead of becoming a duplicate write after eviction. Observing a new active generation clears old state and invalidates captured old keys.
Retention advances only after an entire batch is accepted. A partially failed batch retains its receipts and applies backpressure to different batches until the same ordered batch is replayed. Admission limits are 65,536 possible derived outputs and 64 MiB of serialized source/group bytes multiplied by the possible sink count; oversized batches are rejected before publication and must be split. These limits bound admitted work, not measured heap memory.
Verification: 317 precompute tests pass, including receipt bounds across 100 completed panes, release of accepted payloads, expired/stale replay rejection, independent materialization receipts, and a real sink batch spanning multiple retention horizons that fails after two accepted writes and retries without duplicates. The batch test also checks backpressure and oversized rejection before writes. Rustfmt and
git diff --checkpass.This stacks on #605. The minimum retry horizon is one materialized window; longer configured state retention extends it. This is in-process deduplication, not durable exactly-once recovery. Uncertain downstream writes still require idempotent storage. No measured performance gain is claimed.