Skip to content

Persist summary coordination decisions - #591

Merged
zzylol merged 1 commit into
mainfrom
feat/durable-summary-coordination-journal
Sep 10, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/durable-summary-coordination-journal

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why

Cross-source maintenance must survive process restarts without duplicating summary payloads, accepting ambiguous producer claims, or publishing two meanings under one concrete SummaryInstanceId.

What

  • Add a durable coordination journal containing SummaryStore references, source watermarks, and publication records.
  • Make SummaryInstanceId the staging and publication primary key.
  • Add checksum-backed payload references, producer-epoch isolation, strict equivocation checks, and a lifetime-held single-writer file lock.

How

The journal stores SummaryStateReference metadata and checksums; summary state remains solely in SummaryStore. Mutations validate a cloned snapshot, write and fsync a temporary file, atomically rename it, sync the parent directory, and then update memory. Opening a journal validates schema and rejects duplicate instance IDs or source-epoch watermarks. Equal watermark sequences must repeat exactly.

Before

Coordination lived in memory, and the first draft duplicated state bytes and allowed the same instance ID to acquire different coordinates through a second row.

After

Restart recovery retains staging and publication decisions without copying payloads. Changed coordinates, lineage, checksum, or state reference under an existing instance ID fail as equivocation, and concurrent writers cannot share a journal path.

Verification

  • cargo check -p data_plane --lib
  • Earlier journal tests passed 5/5 before the review hardening.
  • The hardened focused test rebuild was blocked locally when the shared /mydata filesystem filled again; compilation stopped with No space left on device while rebuilding asap_sketchlib.
  • git diff --check

Limitations

This PR provides durable coordination metadata. The next stack wires keyed multi-source barriers into maintenance execution. Family-specific SummaryJoin remains fail-closed until KMV, Theta, or sampling state exposes a typed join capability.

@zzylol
zzylol force-pushed the feat/durable-summary-coordination-journal branch from 2afe718 to 634ac30 Compare September 10, 2026 19:12
@zzylol
zzylol changed the base branch from feat/summary-completion-contract to main September 10, 2026 19:13

@zzylol zzylol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness/storage-boundary findings:

  1. StagedSummaryInput embeds state_bytes in the coordination metadata journal. This conflicts with the SDS boundary established in SummaryInstance: payload belongs to SummaryStore/storage engine and metadata carries SummaryStateReference. Persist the staged payload in SummaryStore and journal a typed state reference/checksum; otherwise every mutation clones and rewrites all summary payloads and creates a second state store.
  2. Staging identity includes input_lineage. The same (catalog, DAG consumer/input, source, instance_id) with changed coordinates or lineage is therefore inserted as a second record instead of rejected as an equivocation. Key by the stable input/instance identity, then require coordinates, lineage, format, and state reference to match on retry.
  3. Publication idempotence uses full-struct equality. Reusing one output instance_id with different coordinates or lineage appends another publication instead of failing. Key publication by catalog/DAG/sink/instance and reject changed details.
  4. Equal watermark sequence with a different watermark currently overwrites the old record. A sequence number must identify one immutable barrier: equal sequence/equal payload is idempotent; equal sequence/different payload must fail. Only a greater sequence with nondecreasing watermark may advance.
  5. open() validates individual rows but does not reject duplicate staging/publication keys or duplicate source watermark rows, so a hand-edited or crash-recovered ambiguous snapshot is accepted. Validate document-wide uniqueness and consistency.

The producer epoch added after #588 fixes the restart-sequence issue. The constant .tmp path and in-process mutex are safe only under a documented single-process/single-writer invariant; otherwise concurrent process writers can overwrite each other and require a lock/compare-and-swap revision.

@zzylol
zzylol force-pushed the feat/durable-summary-coordination-journal branch from 634ac30 to f66d26b Compare September 10, 2026 19:21
@zzylol
zzylol merged commit b7b56dd into main Sep 10, 2026
1 check passed
@zzylol
zzylol deleted the feat/durable-summary-coordination-journal branch September 10, 2026 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant