feat: execute MetricsQL through the shared QueryPlan - #553
Conversation
zzylol
left a comment
There was a problem hiding this comment.
Blocking architecture/correctness issue: MetricsQlPlanCatalog is a language-specific sidecar parallel to the shared QueryPlan catalog.
control_plane/src/metricsql_plan.rs duplicates plan identity, entry lookup, and validation; ExecutableQueryPlan::execution_view clones the DAG back into a PromQL-shaped QueryPlanEntry; and ActivePhysicalPlan gains another catalog field. The sidecar's install validation checks window_ms but does not enforce the materialization's pane_origin_ms, so a MetricsQL binding can pass installation and then fail the strict phase check at read time.
Please make query identity language-tagged in the shared QueryPlan entry/catalog and publish MetricsQL entries in the existing atomic PhysicalPlanInstallRequest. Reuse QueryPlan::validate_against_catalog for the complete binding contract, including pane phase and item labels. The VictoriaMetrics protocol adapter and exact fallback can remain separate. This draft is currently conflicting with main and is not merge-ready.
df000e4 to
4c9c902
Compare
Before this PR
ASAPQuery-backend exposed Prometheus/PromQL only. VictoriaMetrics clients had no independent HTTP surface, tenant-prefixed cluster routes, MetricsQL frontend identity, exact VictoriaMetrics fallback, or way to publish MetricsQL workloads into the authoritative SDS physical-plan lifecycle.
After this PR
/select/{tenant}/prometheus/api/v1/query{,_range}routes, Prometheus-compatible result formatting, and exact fallback that preserves the original MetricsQL request and upstream response status/headers.5f37a850b367f522c7a4bda25d1727e8b44e73eeand lowers supported MetricsQL ASTs into canonicalQueryExprbefore shared physical selection and compilation.QueryPlan; stage/activate, ACK, hot reload, SDS descriptor validation, SummaryStore lookup, and shared DAG execution use the existing physical snapshot lifecycle.sum(rate(...))andsum(increase(...)): real VictoriaMetrics differential testing showed VM extrapolation results differ from the current shared executor. These requests route to exact VictoriaMetrics fallback.readout_lookback_ms=5000survives JSON publication, and compares the published shared-DAG result with VictoriaMetrics.Verification
cargo test -p control_plane metricsql_non_equivalent_counter_rollups_fail_closed_before_publication --lib -q— passedVICTORIAMETRICS_URL=http://127.0.0.1:18428 cargo test -p data_plane --test victoriametrics_accelerated_e2e -- --ignoredagainstvictoriametrics/victoria-metrics:v1.126.0— passedsum(sum_over_time(m[5s])): published DAG,x-asap-execution: warm, backend26, VictoriaMetrics26sum(increase(m[5s])):x-asap-execution: exact_fallback, backend response equals VictoriaMetrics4cargo test -p data_plane --test all_sketches_process_oracle_e2e -- --nocapture— previously passed 4/4 (KLL, HLL, CMS, CountSketch); the final rerun reached the linker but this host ran out of disk andlldterminated with SIGBUSgit diff --check— passedLimitations
MetricsQL acceleration is a proven subset. Parser/lowering, physical publication, missing plan/binding, coverage, and runtime failures fail closed to VictoriaMetrics with a typed stage/reason.
sum(rate(...))andsum(increase(...))are intentionally exact-fallback until the shared executor implements VictoriaMetrics extrapolation. MetricsQL modifiers and parser forms rejected by the pinned frontend also fall back. The shared PromQL types, PromQL lookup contract, and SDS descriptor semantics are unchanged.