Skip to content

feat: execute MetricsQL through the shared QueryPlan - #553

Merged
zzylol merged 5 commits into
mainfrom
feat/victoriametrics-support
Sep 10, 2026
Merged

zzylol merged 5 commits into
mainfrom
feat/victoriametrics-support

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Before this PR

ASAPQuery-backend exposed Prometheus/PromQL only. VictoriaMetrics clients had no independent HTTP surface, tenant-prefixed cluster routes, MetricsQL frontend identity, exact VictoriaMetrics fallback, or way to publish MetricsQL workloads into the authoritative SDS physical-plan lifecycle.

After this PR

  • Adds independent VictoriaMetrics instant/range HTTP adapters, /select/{tenant}/prometheus/api/v1/query{,_range} routes, Prometheus-compatible result formatting, and exact fallback that preserves the original MetricsQL request and upstream response status/headers.
  • Pins ASAPPlanner 5f37a850b367f522c7a4bda25d1727e8b44e73ee and lowers supported MetricsQL ASTs into canonical QueryExpr before shared physical selection and compilation.
  • Publishes MetricsQL-tagged entries in the authoritative QueryPlan; stage/activate, ACK, hot reload, SDS descriptor validation, SummaryStore lookup, and shared DAG execution use the existing physical snapshot lifecycle.
  • Keeps language identity at the boundary: MetricsQL lookup cannot match a PromQL-tagged entry with identical text, and existing PromQL lookup behavior remains intact.
  • Extends shared physical bindings for selected per-series sketch materializations and process-level KLL/HLL/CMS/CountSketch readout without changing SDS descriptor meaning.
  • Fails closed before publication for sum(rate(...)) and sum(increase(...)): real VictoriaMetrics differential testing showed VM extrapolation results differ from the current shared executor. These requests route to exact VictoriaMetrics fallback.
  • Adds a real VictoriaMetrics E2E that imports nine samples into VictoriaMetrics, sends the same samples through backend Remote Write/precompute/SummaryStore, installs the compiled MetricsQL plan, verifies readout_lookback_ms=5000 survives JSON publication, and compares the published shared-DAG result with VictoriaMetrics.

Verification

  • cargo test -p control_plane metricsql_non_equivalent_counter_rollups_fail_closed_before_publication --lib -q — passed
  • VICTORIAMETRICS_URL=http://127.0.0.1:18428 cargo test -p data_plane --test victoriametrics_accelerated_e2e -- --ignored against victoriametrics/victoria-metrics:v1.126.0 — passed
    • sum(sum_over_time(m[5s])): published DAG, x-asap-execution: warm, backend 26, VictoriaMetrics 26
    • sum(increase(m[5s])): x-asap-execution: exact_fallback, backend response equals VictoriaMetrics 4
  • cargo test -p data_plane --test all_sketches_process_oracle_e2e -- --nocapture — previously passed 4/4 (KLL, HLL, CMS, CountSketch); the final rerun reached the linker but this host ran out of disk and lld terminated with SIGBUS
  • git diff --check — passed

Limitations

MetricsQL acceleration is a proven subset. Parser/lowering, physical publication, missing plan/binding, coverage, and runtime failures fail closed to VictoriaMetrics with a typed stage/reason. sum(rate(...)) and sum(increase(...)) are intentionally exact-fallback until the shared executor implements VictoriaMetrics extrapolation. MetricsQL modifiers and parser forms rejected by the pinned frontend also fall back. The shared PromQL types, PromQL lookup contract, and SDS descriptor semantics are unchanged.

@zzylol zzylol left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking architecture/correctness issue: MetricsQlPlanCatalog is a language-specific sidecar parallel to the shared QueryPlan catalog.

control_plane/src/metricsql_plan.rs duplicates plan identity, entry lookup, and validation; ExecutableQueryPlan::execution_view clones the DAG back into a PromQL-shaped QueryPlanEntry; and ActivePhysicalPlan gains another catalog field. The sidecar's install validation checks window_ms but does not enforce the materialization's pane_origin_ms, so a MetricsQL binding can pass installation and then fail the strict phase check at read time.

Please make query identity language-tagged in the shared QueryPlan entry/catalog and publish MetricsQL entries in the existing atomic PhysicalPlanInstallRequest. Reuse QueryPlan::validate_against_catalog for the complete binding contract, including pane phase and item labels. The VictoriaMetrics protocol adapter and exact fallback can remain separate. This draft is currently conflicting with main and is not merge-ready.

@zzylol
zzylol force-pushed the feat/victoriametrics-support branch from df000e4 to 4c9c902 Compare September 10, 2026 06:10
@zzylol zzylol changed the title feat: add VictoriaMetrics MetricsQL query support feat: execute MetricsQL through the shared QueryPlan Sep 10, 2026
@zzylol
zzylol marked this pull request as ready for review September 10, 2026 06:16
@zzylol
zzylol merged commit 1abcaa6 into main Sep 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant