Skip to content

feat: compose ClickHouse exact subtrees with summaries - #577

Closed
zzylol wants to merge 4 commits into
mainfrom
feat/unified-clickhouse-external-leaf
Closed

zzylol wants to merge 4 commits into
mainfrom
feat/unified-clickhouse-external-leaf

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why

A ClickHouse SQL query could previously use either an ASAP summary plan or whole-query ClickHouse fallback. It could not execute one planned subtree from SummaryStore, execute another exact subtree in ClickHouse, and compose both through the published query DAG.

What

  • Add a ClickHouse-only ExternalSqlLeaf to the single published QueryPlan DAG.
  • Compile planner-selected exact cuts from explicit, fingerprinted workload artifacts.
  • Prefetch external leaves asynchronously and compose their typed relations with SummaryStore readouts through relational join and project nodes.
  • Validate SQL, parameters, schemas, catalog-derived summary output, and exact-cut consumption before publication.
  • Decode ClickHouse JSONCompact timestamps, including timezone-qualified DateTime64.
  • Document the explicit exact-subtree artifact contract and fail-closed boundaries.

How

The control plane matches every external artifact against the complete canonical KeepPreAsap subtree and its fingerprint. The data plane snapshots one active physical generation, fetches ClickHouse leaves, reads summary leaves by SummaryDefinitionId, and executes the parent relational DAG locally. Missing artifacts, incomplete coverage, schema drift, external errors, or unsupported shapes fall back to the original whole SQL query.

Before this PR

A ratio query with one accelerated input and one unsupported input fell back entirely to ClickHouse.

After this PR

The accelerated input is read from the real SummaryStore, the unsupported input is executed as an exact ClickHouse subtree, and ASAPQuery joins both typed relations and evaluates the final projection locally in the same QueryPlan DAG.

Verification

  • cargo check -p control_plane -p data_plane --all-targets
  • Production planner exact-cut compile/publication test
  • Catalog-derived mixed-schema positive and tampering tests
  • Real SummaryStore + real ClickHouse external leaf + local Join/Project E2E
  • Real ClickHouse timezone-qualified DateTime64 typed decode test
  • Independent agent review: PASS

Human review

Reserved for human reviewers.

@zzylol

zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #589, which ports the typed ClickHouse exact-leaf and mixed SummaryStore join/project behavior into the shared language-neutral QueryPlan DAG without the former SQL-specific plan/catalog layers.

@zzylol zzylol closed this Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant