Persist summary coordination decisions - #591
Merged
Merged
Conversation
zzylol
force-pushed
the
feat/durable-summary-coordination-journal
branch
from
September 10, 2026 19:12
2afe718 to
634ac30
Compare
zzylol
changed the base branch from
feat/summary-completion-contract
to
main
September 10, 2026 19:13
zzylol
commented
Sep 10, 2026
zzylol
left a comment
Contributor
Author
There was a problem hiding this comment.
Correctness/storage-boundary findings:
StagedSummaryInputembedsstate_bytesin the coordination metadata journal. This conflicts with the SDS boundary established inSummaryInstance: payload belongs to SummaryStore/storage engine and metadata carriesSummaryStateReference. Persist the staged payload in SummaryStore and journal a typed state reference/checksum; otherwise every mutation clones and rewrites all summary payloads and creates a second state store.- Staging identity includes
input_lineage. The same(catalog, DAG consumer/input, source, instance_id)with changed coordinates or lineage is therefore inserted as a second record instead of rejected as an equivocation. Key by the stable input/instance identity, then require coordinates, lineage, format, and state reference to match on retry. - Publication idempotence uses full-struct equality. Reusing one output
instance_idwith different coordinates or lineage appends another publication instead of failing. Key publication by catalog/DAG/sink/instance and reject changed details. - Equal watermark sequence with a different watermark currently overwrites the old record. A sequence number must identify one immutable barrier: equal sequence/equal payload is idempotent; equal sequence/different payload must fail. Only a greater sequence with nondecreasing watermark may advance.
open()validates individual rows but does not reject duplicate staging/publication keys or duplicate source watermark rows, so a hand-edited or crash-recovered ambiguous snapshot is accepted. Validate document-wide uniqueness and consistency.
The producer epoch added after #588 fixes the restart-sequence issue. The constant .tmp path and in-process mutex are safe only under a documented single-process/single-writer invariant; otherwise concurrent process writers can overwrite each other and require a lock/compare-and-swap revision.
zzylol
force-pushed
the
feat/durable-summary-coordination-journal
branch
from
September 10, 2026 19:21
634ac30 to
f66d26b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Cross-source maintenance must survive process restarts without duplicating summary payloads, accepting ambiguous producer claims, or publishing two meanings under one concrete SummaryInstanceId.
What
How
The journal stores
SummaryStateReferencemetadata and checksums; summary state remains solely in SummaryStore. Mutations validate a cloned snapshot, write and fsync a temporary file, atomically rename it, sync the parent directory, and then update memory. Opening a journal validates schema and rejects duplicate instance IDs or source-epoch watermarks. Equal watermark sequences must repeat exactly.Before
Coordination lived in memory, and the first draft duplicated state bytes and allowed the same instance ID to acquire different coordinates through a second row.
After
Restart recovery retains staging and publication decisions without copying payloads. Changed coordinates, lineage, checksum, or state reference under an existing instance ID fail as equivocation, and concurrent writers cannot share a journal path.
Verification
cargo check -p data_plane --lib/mydatafilesystem filled again; compilation stopped withNo space left on devicewhile rebuildingasap_sketchlib.git diff --checkLimitations
This PR provides durable coordination metadata. The next stack wires keyed multi-source barriers into maintenance execution. Family-specific SummaryJoin remains fail-closed until KMV, Theta, or sampling state exposes a typed join capability.