Skip to content

Persist executable DAG bindings in installed plans - #579

Merged
zzylol merged 1 commit into
mainfrom
refactor/persist-post-asap-dag
Sep 10, 2026
Merged

zzylol merged 1 commit into
mainfrom
refactor/persist-post-asap-dag

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why

The compiler translated Planner post-ASAP DAGs into query and precompute projections, then discarded the typed semantic DAG and backend placement map. The installed plan therefore could not schedule maintenance subDAGs or prove which Planner node produced each materialization/query node. Hybrid exact work was also encoded as a PromQL-specific logical side channel instead of a normal query DAG dependency.

Before

  • PrecomputePlan retained materializations but not the Planner DAG or backend node placement.
  • Runtime code had to reconstruct execution intent from materialization/query shapes.
  • Candidate TopK used LogicalOperator::CandidateExactSubquery, coupling the shared QueryPlan DAG to PromQL preparation.

After

  • Persist a versioned PostAsapDagDocument plus BackendExecutableBinding for every compiled query in PrecomputePlan and therefore in the immutable ActivePhysicalPlan snapshot.
  • Preserve independent PostAsapNodeId, SummaryDefinitionId, and QueryNodeId domains, including explicit maintenance/query placement and sink references.
  • Validate document schema, typed payload/operator agreement, complete placement, materialization sinks, and QueryPlan references before publication.
  • Add language-neutral QueryPlanNode::ExternalExact with an explicit output shape and input contracts. Hybrid Candidate TopK now consumes candidate membership and external exact values through ordinary DAG edges.
  • Keep the installed wire document Send + Sync by storing Planner payload/schema JSON and decoding it immediately into Planner-owned typed values before execution.

Example

For hybrid topk(2, sum by(job)(rate(m[1m]))), the installed query DAG is:

CMS membership readout -> ExternalExact(candidate_membership=job) -> CandidateTopK

The exact engine receives only the selected membership set and returns an instant vector; CandidateTopK combines both dependency results normally.

Verification

  • cargo check --workspace --all-targets
  • cargo test -p control_plane --lib physical::compiler::tests (67 passed)
  • cargo test -p control_plane --lib query_plan::tests (5 passed)
  • cargo test -p data_plane --lib precompute_engine::subdag_scheduler::tests

Dependency and migration surface

Depends on #576. Runtime scheduler work should consume PrecomputePlan.executable_dags; external engines implement ExternalExactRequest and return the declared ExternalExactOutput. ClickHouse integration should map its exact leaf to this shared node and must not add a parallel plan catalog or executable-plan envelope.

@zzylol
zzylol force-pushed the refactor/persist-post-asap-dag branch 3 times, most recently from fa050c8 to affc4ba Compare September 10, 2026 16:41
@zzylol
zzylol force-pushed the refactor/persist-post-asap-dag branch from affc4ba to 6832db8 Compare September 10, 2026 16:48
@zzylol
zzylol changed the base branch from refactor/unify-query-language to main September 10, 2026 16:49
@zzylol
zzylol merged commit 0e0519e into main Sep 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant