Skip to content

Schedule complete finite source cohorts through immutable maintenance - #667

Merged
zzylol merged 19 commits into
mainfrom
feat/maintenance-finite-cohort
Sep 11, 2026
Merged

zzylol merged 19 commits into
mainfrom
feat/maintenance-finite-cohort

Conversation

@zzylol

@zzylol zzylol commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Why

A derived maintenance program must wait for every source at the same window. Publishing one source or pairing different windows would materialize a partial result.

What

The finite scheduler can route a complete cohort of raw input definitions into the existing immutable maintenance executor. Each source must have exactly one physical SID and group, all explicit groups must match, and windows must be aligned, stored in full, and non-overlapping.

How

Use the shared source-window validator, inspect durable population inventories, and select the full window intersection. Prove every requested source before resolving an output, then reuse the existing all-source lifetime transaction, lineage digest, and durable retry path. Single- and multi-source scheduling pass their captured catalog generation through execution instead of refreshing it at the publication boundary.

Before this PR

The public finite scheduler rejected multiple source definitions, even though the immutable cohort executor could execute a bound multi-frontier program.

After this PR

A bound two-source fixture with completed values 2 and 7 produces one durable result 9 and reuses it after repeat scheduling and restart. If the sources complete different windows, no target SID, metadata receipt, or part is created. A stale generation is rejected before another generation's state can be substituted.

Verification

  • Actual sketchlib refactor(legacy_expr): step β — thread Schema through legacy planning stack #139: all 13 maintenance-runtime tests pass (3.98 s), including the scheduler/restart, disjoint-window zero-output, and captured-generation regressions.
  • Independent source review passed for exact production commit 64079926.
  • Integration head 675f9d1b adds only the previously reviewed shared population codec and explicit ReadTime in an offline fixture; maintenance source is identical.
  • The preceding timed-operator layer passed 338 precompute and 2 exact binary tests.
  • The fixture is a bound operator/scheduler test, not evidence of actual Planner-selected automatic multi-source HTTP execution. That integration is the next layer.

Limits

No group shuffle, continuous watermark, sliding execution, or new-generation seal liveness claim. Restored historical writes without a new generation's completion proof remain conservatively rejected. Compiler/install capability enablement and the real selected-plan process test remain separate.

Stacked on #666 and depends on Planner #402.

@zzylol
zzylol marked this pull request as ready for review September 11, 2026 19:03
@zzylol
zzylol changed the base branch from feat/maintenance-timed-binary to main September 11, 2026 19:15
@zzylol
zzylol merged commit 0f41675 into main Sep 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant