Skip to content

Evaluate 27 ClickHouse SQL queries on current main - #595

Merged
zzylol merged 2 commits into
mainfrom
eval/sql-27-main
Sep 10, 2026
Merged

zzylol merged 2 commits into
mainfrom
eval/sql-27-main

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

The merged ClickHouse shared-DAG path had not been evaluated against the 27-query o11y SQL corpus on current main, so the old #560 result could no longer serve as acceptance evidence. This PR adds a reproducible current-main audit and records the resulting parser → Planner → publication → data-plane routing matrix.

The run is pinned to 791f7d7b (which contains #589/#588/#592) and reuses only the corpus input. It invokes the production SQL frontend and post-ASAP selector, attempts a validated unified QueryPlan publication for every plannable query, installs a current-generation catalog in the data plane, and asks the catalog-backed ClickHouse accelerator to route all 27 requests.

Before this PR, the latest checked-in result described the removed sidecar architecture and reported 27 exact fallbacks without current pipeline evidence. After this PR, tools/o11y-sql-main-eval/artifacts/matrix.json records 0 warm, 0 partial hybrid, 27 exact fallback, and 0 request failures. Twenty-four queries fail closed in the ClickHouse SQL frontend. q05, q06, and q23 reach post-ASAP MinMax plans but publication rejects their unbound metric population predicate, after which the data plane returns CatalogMiss and uses exact fallback. All 27 were sent through the production ClickHouse HTTP listener to a live ClickHouse 26.8.2.7 backend: every response was HTTP 200 and byte-for-byte equal to a direct exact request. Artifacts distinguish fallback_requested, exact_executed, exact_success, and direct-response equality.

The report also audits IDs, schemas, bindings, and execution nodes. Current production uses one QueryPlanNode, QueryNodeId, MaterializationBinding, SummaryDefinitionId, SummarySchema, and backend QueryLanguage; no SQL sidecar catalog or parallel activation lifecycle remains. SQL-specific structures are protocol/planning context adapters around the shared plan. Hand-built alternate plans are confined to tests.

Verification:

  • tools/o11y-sql-main-eval/reproduce.sh
  • cargo test -q -p control_plane clickhouse (1 passed)
  • cargo test -q -p data_plane query_engines::asap_clickhouse_query_engine (12 passed)
  • cargo fmt --check
  • git diff --check

Limitations: no query in this corpus can currently be published warm or hybrid, so this PR does not claim accelerated-versus-exact value equality or performance benefit. The report identifies the exact frontend and SDS population-binding gaps that prevent that experiment.

@zzylol
zzylol merged commit c0ab7ef into main Sep 10, 2026
1 check passed
@zzylol
zzylol deleted the eval/sql-27-main branch September 10, 2026 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant