Skip to content

Choose costed sliding-window materialization layouts - #580

Merged
zzylol merged 1 commit into
mainfrom
feat/sliding-window-layout-candidates
Sep 10, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/sliding-window-layout-candidates

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

A semantic sliding window was previously compiled only as non-overlapping panes, and the compiler overloaded window_size with the chosen pane width. That made full overlapping windows impossible to cost or install and blurred query semantics with storage layout.

This change makes the backend choose among typed physical layouts while keeping the Planner DAG semantic:

  • adds catalog-visible Pane, FullWindow, and exact HierarchicalRollup layouts;
  • keeps window_size, slide_interval, and pane_size independent;
  • includes the physical layout in SummaryDefinitionId identity and catalog bindings;
  • validates layout divisibility and framework compatibility before planning or installation;
  • passes every concrete implementation's measured weighted cost to Planner, with auditable CPU, memory, storage, network, and scan components;
  • computes retention according to the selected layout;
  • emits the selected layout to precompute and collector plans without stringly typed state_layout fields.

For a 60-second window evaluated every 10 seconds, the same post-ASAP DAG can now choose either six 10-second panes or one complete 60-second state produced every 10 seconds. Tests demonstrate selection of the full window when its query-frequency-adjusted cost is lower, selection of panes when update/storage cost dominates, multiple valid slide frequencies, and fail-closed non-divisible layouts.

Validation:

  • cargo test -p control_plane --lib --quiet (712 passed)
  • cargo test -p asap_types --quiet (70 passed)
  • cargo check -p data_plane
  • git diff --check

Stacked on #579 because the installed precompute plan must retain the executable DAG bindings alongside this physical layout.

@zzylol
zzylol force-pushed the refactor/persist-post-asap-dag branch from c203334 to 8ee7f98 Compare September 10, 2026 16:38
@zzylol
zzylol force-pushed the feat/sliding-window-layout-candidates branch from 9d7dc9a to 886db4d Compare September 10, 2026 16:39
@zzylol
zzylol force-pushed the refactor/persist-post-asap-dag branch 3 times, most recently from affc4ba to 6832db8 Compare September 10, 2026 16:48
@zzylol
zzylol force-pushed the feat/sliding-window-layout-candidates branch from 886db4d to a587572 Compare September 10, 2026 17:13
@zzylol
zzylol changed the base branch from refactor/persist-post-asap-dag to main September 10, 2026 17:13
@zzylol
zzylol merged commit b6fcf06 into main Sep 10, 2026
1 check passed
zzylol added a commit that referenced this pull request Sep 10, 2026
* Add costed sliding-window materialization layouts (#580)

* Execute selected sliding window layouts in workers (#583)

* Execute ClickHouse joins in the shared query DAG (#569)

* feat(clickhouse): execute SQL joins in the shared query DAG

* fix: consume canonical planner lifecycle contract

* fix(clickhouse): validate each join input coverage

* chore: pin merged planner join contract
@zzylol
zzylol deleted the feat/sliding-window-layout-candidates branch September 21, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant