Skip to content

refactor: split backend plans and remove Collector dependency - #749

Draft
zzylol wants to merge 3 commits into
mainfrom
refactor/backend-plan-split
Draft

zzylol wants to merge 3 commits into
mainfrom
refactor/backend-plan-split

Conversation

@zzylol

@zzylol zzylol commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Why

The backend persisted complete selected Planner DAGs under PrecomputePlan.executable_dags, including query-only operators, and fetched asap-precompute-rs from ASAPCollector to decode sketch state. The design in #737 separates maintenance from query execution and removes that build/runtime dependency.

Before this PR

For Input → BuildSummary → SummaryEstimate → Result, PrecomputePlan contained all four semantic nodes. The backend reconstructed DDSketch/KLL through a Collector wrapper and some tests required Collector's Rust crate.

After this PR

  • The compiler emits a version 2 maintenance projection ending at stored summary sinks. QueryPlan owns the executable readout graph and retains the complete selected DAG only as provenance. PromQL and ClickHouse SQL use the same ownership rule.
  • Publication and backend staging validate writer/query identity and provenance together. Supported version 1 complete-DAG artifacts normalize to the split runtime representation before activation; exact-only legacy DAGs leave no PrecomputePlan executable content.
  • A neutral asap_sketch_codec crate decodes full and legacy bare sketchlib envelopes. Backend ingest and tests use sketchlib directly. asap-precompute-rs and the ASAPCollector Cargo patch are removed from the backend dependency graph.

Verification

  • cargo test -p asap_types --locked --quiet: 108 passed.
  • cargo test -p control_plane --locked --quiet: 438 passed across targets.
  • cargo test -p data_plane --lib --locked --quiet: 1,182 passed; the new legacy-normalization test was rerun after its final change.
  • cargo test -p data_plane --test edge_sketch_codec --locked -- --test-threads=1: 4 passed.
  • cargo test -p data_plane --test asapquery_compatibility_process_e2e --locked -- --test-threads=1: 17 passed, 1 existing stale-evidence test ignored; a production process test was rerun after the normalization change.
  • ASAP_E2E_CARGO_TARGET_DIR="$PWD/target" ./scripts/e2e.sh whole: passed controller publication → CollectorPlan exchange → OTLP ingest → PromQL serving and failed-rollout preservation.
  • cargo tree -p data_plane --locked contains no ASAPCollector or asap-precompute-rs; formatting and git diff --check pass.

Remaining migration work

This draft does not yet implement the SDS state-slot/instance contract and standalone catalog-materialization removal in #737. The version 1 reader remains for compatibility; retirement of the old execution path requires the remaining migration gates. One older ignored ERP/KLL process fixture still needs refreshed lifecycle evidence; the whole-backend process E2E is now active and passes.

@zzylol zzylol changed the title refactor: split installed maintenance DAGs from query execution refactor: split backend plans and remove Collector dependency Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant