feat(precompute): execute immutable summary subDAGs - #656
Merged
Merged
Conversation
zzylol
changed the base branch from
feat/immutable-output-publication
to
main
September 11, 2026 16:13
zzylol
marked this pull request as ready for review
September 11, 2026 16:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this PR, maintenance execution could merge states, but could not finalize several immutable source windows and feed their results into another summary without treating correction fragments as fresh observations.
After this PR, an explicit precompute entry point executes the installed subDAG over frozen SummaryStore windows. It preserves pane collections until explicit merge/finalize/aggregate operators, validates the derived input identity, and uses the existing flusher's durable publication protocol. A stored lineage lookup avoids rebuilding randomized sketches on retry. The store handles state/provenance and publication; DAG execution stays in the precompute engine.
Verification: nine maintenance tests and all 311 SketchStore tests passed on the integrated main branch with the pinned sketch library. The real Sum → Finalize → KLL test covers failed manifest publication, recovery of the pending durable part after restart, latest-result retry with one output part, and late-correction rejection. An Int64 schema is explicitly rejected by the Float64-only finalizer. Performance measurements are not applicable.
This is an execution primitive, not automatic workload coverage. Derived installation remains guarded until compiler construction and scheduling are connected. The current consumer supports one source/population, complete non-overlapping panes, Float64 Sum/Count finalization, and unkeyed updates. General row operators, multiple-source shuffles, overlapping replacement, continuous watermarks, and historical metadata GC remain follow-up work.
The durable publication prerequisite (#654) is merged. Independent source review passed after the Float64 schema, pending recovery, and additive-admission fixes. Current-head CI remains the final merge gate.