Define source-scoped summary completion barriers - #588
Merged
Merged
Conversation
zzylol
commented
Sep 10, 2026
zzylol
left a comment
Contributor
Author
There was a problem hiding this comment.
Correctness findings before the coordination journal builds on this contract:
OutputSink::advance_summary_watermarkdefaults toOk(())while discarding the barrier. A producer can therefore treat a completeness claim as accepted even when the sink cannot persist or forward it. The default should return an explicit unsupported error, or support must be negotiated as a capability.- Monotonic
sequenceis scoped only by(CatalogGeneration, producer_id, partition_id), but the contract does not defineproducer_idas a restart incarnation. A restarted producer can reuse sequence numbers under the same catalog generation. Add a producer epoch/incarnation or make it an explicit invariant in a validated typed identity. - Add validation/tests for nonempty source components, a nonempty half-open window, retry-stable nonempty lineage, and malformed serde input. The next journal must reject regressions and equal-sequence/different-payload conflicts rather than relying on these structs alone.
SummaryWindowIdentityrepeats the coordinate fields already present inSummaryInstance. Please make the relationship explicit: either rename it toSummaryInstanceCoordinatesand derive/use the stableSummaryInstanceId, or carry that instance ID in completion records. Otherwise storage inventory and completion idempotency can create two identities for the same state.
The source-partition event-time watermark model is otherwise the right boundary, provided downstream closes only windows whose end is covered by the barrier.
zzylol
force-pushed
the
feat/summary-completion-contract
branch
from
September 10, 2026 18:46
0db9a2f to
8f4d360
Compare
This was referenced Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Cross-source maintenance needs an explicit proof that one ordered producer partition has delivered every window through an event-time boundary. The worker's diagnostic maximum timestamp cannot provide that proof, and silently accepting a barrier in a sink that does not implement coordination can publish incomplete state.
What
SummaryInstanceIdfor the concrete SDS instance and keep time/group data asSummaryInstanceCoordinates.Unsupported.How
Completion records bind catalog generation, source incarnation, concrete instance ID, half-open time range, named group coordinates, and retry-stable lineage. Validation rejects empty sources and lineage, zero epochs and sequences, invalid ranges, and invalid catalog identities. Wire structs deny unknown fields so incompatible producers fail closed.
Before
Maintenance code could infer completion from emitted payloads or unrelated worker timestamps. A sink could also discard a watermark through the trait default while reporting success.
After
A coordinator can distinguish producer restarts, reject malformed completion claims, and negotiate barrier support explicitly before using source watermarks.
Verification
cargo test -p asap_types sds::tests— 17 passedgit diff --checklldexited with SIGBUS. The worktree target was cleaned; CI provides the clean-environment result.Limitations
This PR defines the shared boundary. The next stacked change persists staged inputs, barriers, and atomic publication records for restart-safe coordination.