Skip to content

feat: all-five-sketch query_statistic + cold-store wiring (P1) - #69

Merged
zzylol merged 1 commit into
mainfrom
feat/all-five-sketch-query-statistic
May 1, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/all-five-sketch-query-statistic

Conversation

@zzylol

@zzylol zzylol commented May 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Backend half of the runtime e2e: each sketch type now has a verified PromQL → backend path through the modified-OTLP wire format (typed Metric.data = {DDSketch | KLLSketch | HLLSketch | CountSketch | CountMinSketch}). Companion PR in ASAPCollector (#204) lands the agent + harness.

query_statistic implementations

Accumulator Statistics Notes
DDSketchAccumulator Quantile / Sum / Count / Min / Max
HllSketchAccumulator Cardinality (Count accepted as alias) lets the existing PromQL count(...) path land on HLL
CountSketchAccumulator Topk / Count / Sum no-key fallback returns row-mean total
CountMinSketchAccumulator Count / Sum no-key fallback returns min-row sum — canonical CMS total-event estimator, exact when each insert increments one cell per row

Factory wiring

accumulator_factory.rs: DDSketchAccumulatorUpdater wired in alongside CMS / CountSketch / KLL / HLL updaters so the precompute_engine recognises AggregationType::DDSketch from streaming.yaml.

Modified-OTLP envelope decoders go through from_sketchlib_proto_bytes / from_msgpack_bytes against SketchEnvelope { sketch_state: Some(SketchState::*(state)) }.

P1 — cold-store flag

asap-query-engine/src/main.rs:

  • --cold-store-root with env = "ASAP_COLD_STORE_ROOT" plumbed through a build_adapter_config helper that selects prometheus_promql_with_cold when set.
  • Combine with --forward-unsupported-queries to keep Prom as the tail of the chain; without it, unsupported shapes return empty.
  • Four unit tests pin the wiring matrix (cold × forward).

This unblocks the three-way harness (#66) reporting the cold-tier accuracy figure end-to-end, and the cold path is what carries the e2e flow today (warm-tier sketch ingest is dropped at the OTel gateway — see ASAPCollector#204 follow-ups).

Known cleanup (not a blocker)

compatible_agg_types in capability_matching.rs doesn't list CountMinSketch under Statistic::Sum, but query_logics treats CMS as the canonical approximator for both Sum and Count. The exact-match find_query_config path bypasses capability matching, so the runtime e2e succeeds today; reconciling the two tables is a separate change.

Test plan

  • cargo build / cargo check clean
  • All five sketches round-trip through agent (sketchlib-go) → wire → backend accumulator
  • DDSketch: histogram_quantile(0.5/0.9/0.99,…) → 21.12 / 47.95 / 104.60 within ε=0.01 (relative_quantile)
  • KLLSketch: histogram_quantile(0.5,…) → 18.26 within ε=0.16 (rank_quantile)
  • HLLSketch: count(http_requests_total) → 149.68 within ε=0.008 (relative_cardinality)
  • CountSketch: sum_over_time(http_requests_total[1m]) → 24266 within ε=0.03 (additive_frequency)
  • CountMinSketch: sum_over_time(http_requests_total[1m]) → 145735 within ε≈0.0027, δ=0.03125
  • P1 unit tests: cold-store-root × forward-unsupported (4 cases)

🤖 Generated with Claude Code

Lands the backend half of the runtime e2e: each sketch type now has
a verified PromQL → backend path through the modified-OTLP wire
format (typed Metric.data = {DDSketch | KLLSketch | HLLSketch |
CountSketch | CountMinSketch}).

query_statistic implementations:
- DDSketchAccumulator: Quantile / Sum / Count / Min / Max.
- HllSketchAccumulator: Cardinality, with Count accepted as a
  Cardinality alias for the existing PromQL count(...) path.
- CountSketchAccumulator: Topk / Count / Sum, no-key fallback
  returns row-mean total.
- CountMinSketchAccumulator: Count / Sum, no-key fallback returns
  the min-row sum — canonical CMS total-event estimator that's
  exact when each insert increments one cell per row (and CMS
  never under-counts).

accumulator_factory.rs: DDSketchAccumulatorUpdater wired in
alongside CMS / CountSketch / KLL / HLL updaters so the
precompute_engine recognises AggregationType::DDSketch from
streaming.yaml.

Modified-OTLP envelope decoders go through
from_sketchlib_proto_bytes / from_msgpack_bytes against
SketchEnvelope { sketch_state: Some(SketchState::*(state)) }.

P1 — cold-store flag wiring (asap-query-engine/main.rs):
- --cold-store-root with env = "ASAP_COLD_STORE_ROOT" plumbed
  through a build_adapter_config helper that selects
  prometheus_promql_with_cold when set.
- Combine with --forward-unsupported-queries to keep Prom as the
  tail of the chain.
- Four unit tests pin the wiring matrix (cold × forward).

Companion PR in ASAPCollector lands the agent + harness
(see ProjectASAP/ASAPCollector#204).

Known cleanup (not a blocker): compatible_agg_types in
capability_matching.rs doesn't list CountMinSketch under
Statistic::Sum, but query_logics treats CMS as the canonical
approximator for both Sum and Count. The exact-match
find_query_config path bypasses capability matching, so the
runtime e2e succeeds today; reconciling the two tables is a
separate change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 682741b into main May 1, 2026
@zzylol
zzylol deleted the feat/all-five-sketch-query-statistic branch May 1, 2026 03:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant