Skip to content

fix: bound maintenance replay state by materialization lifecycle - #607

Merged
zzylol merged 2 commits into
mainfrom
fix/bound-maintenance-commit-state
Sep 11, 2026
Merged

zzylol merged 2 commits into
mainfrom
fix/bound-maintenance-commit-state

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Maintenance replay state retained every serialized input and derived payload after publication, independently of SummaryStore retention. Its query-local node IDs could also collide across installed DAGs.

Before this PR, completed panes accumulated for the process lifetime and unrelated materializations could share a publication key. After this PR, receipts use target summary-definition identity and SHA-256 input lineage, accepted payloads are released, and receipt retention follows the installed materialization's event-time retention span. Expired replay is rejected instead of becoming a duplicate write after eviction. Observing a new active generation clears old state and invalidates captured old keys.

Retention advances only after an entire batch is accepted. A partially failed batch retains its receipts and applies backpressure to different batches until the same ordered batch is replayed. Admission limits are 65,536 possible derived outputs and 64 MiB of serialized source/group bytes multiplied by the possible sink count; oversized batches are rejected before publication and must be split. These limits bound admitted work, not measured heap memory.

Verification: 317 precompute tests pass, including receipt bounds across 100 completed panes, release of accepted payloads, expired/stale replay rejection, independent materialization receipts, and a real sink batch spanning multiple retention horizons that fails after two accepted writes and retries without duplicates. The batch test also checks backpressure and oversized rejection before writes. Rustfmt and git diff --check pass.

This stacks on #605. The minimum retry horizon is one materialized window; longer configured state retention extends it. This is in-process deduplication, not durable exactly-once recovery. Uncertain downstream writes still require idempotent storage. No measured performance gain is claimed.

@zzylol
zzylol changed the base branch from fix/maintenance-publication-retry to main September 11, 2026 00:05
@zzylol
zzylol merged commit b47cb9a into main Sep 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant