fix: warm tier ingests + answers HLL count / KLL quantile (and best-effort CountSketch/CountMin)

[zzylol]

Root cause

Issue ProjectASAP/ASAPCollector#46 — agent emits 5 sketch envelopes (DDSketch / KLL / HLL / CountSketch / CountMin) over modified-OTLP, but four families never produced a non-empty warm-tier answer even when the streaming-config registered the matching aggregation. Two backend gaps closed here:

1. HLL _hll alias was not resolved — agent renames unique_users_per_min → unique_users_per_min_hll on egress. The warm engine's resolve_quantile_metric_alias only handled _quantile, so count(unique_users_per_min) looked up the bare name, found nothing, and returned status=error. Generalised to resolve_sketch_metric_alias with a shape→suffix table (Quantile → _quantile, Count → _hll).
2. Statistic::Rate over CMS was the documented PR fix(warm-engine): answer replay queries under empty-schema deploy (was status=error) #111 honest gap — compatible_agg_types(Rate) excluded CMS, and CountMinSketchAccumulator::query_statistic rejected Rate. Wired both: capability now matches CMS for rate(metric[range]), engine pushes range_ms into query_kwargs, accumulator divides min-row-sum by range_ms / 1000 for an events/sec answer (falls back to raw count when range_ms is absent).

OTLP ingest decoder's per-variant dispatch was already in place from PRs C/G; the residual gaps were query-side. Specifically asap-query-engine/src/drivers/ingest/otel.rs::decode_modified_otlp_sketch_bytes already routes HLL / KLL / DDSketch / CountSketch / CountMinSketch envelopes to the matching from_sketchlib_proto_bytes constructor.

Per-family status

Family	Wire decode	Warm engine answers query
DDSketch	yes	yes — quantile_over_time(...) (existing)
KLL	yes	yes — quantile_over_time(...) (capability matched DatasketchesKLL; pinned by new test)
HLL	yes	yes — count(metric) after _hll alias rewrite (new resolver + cap-match + pinned test)
CountMin	yes	yes — rate(metric[range]) (new CMS-Rate cap-match + accumulator support)
CountSketch	yes	partial — topk(K, ...) capability resolves to CountSketch, but the keyed-merge path still needs a paired SetAggregator. Documented as PR #111 open call-out.

Honest-gap call-outs that remain open after this PR

• topk(K, top_endpoint_qps) over CountSketch still requires a paired SetAggregator to surface the keys (the merge path's collect_results_separate_keys walks precompute.get_keys()). Tracked under PR fix(warm-engine): answer replay queries under empty-schema deploy (was status=error) #111.
• MSGPACK_DELTA (encoding=4) for every sketch family is still Err("MSGPACK_DELTA encoding is not yet wired") in decode_modified_otlp_sketch_bytes. Tracked under PR I.

Files changed

• asap-common/dependencies/rs/asap_types/src/capability_matching.rs — compatible_agg_types(Rate) adds CountMinSketch + CountMinSketchWithHeap.
• asap-query-engine/src/precompute_operators/count_min_sketch_accumulator.rs — query_statistic answers Rate (via range_ms from kwargs) and Increase.
• asap-query-engine/src/engines/simple/engine.rs — resolve_sketch_metric_alias (replaces _quantile-only resolver), build_query_kwargs_promql injects range_ms for Rate/Increase, four new test modules.

Test plan

• cargo build --release (clean)
• cargo test --release --lib — 950 passing (baseline 932), 34 pre-existing baseline failures unchanged (verified by git stash + re-test).
• +4 tests in precompute_operators::count_min_sketch_accumulator::tests (Rate with range_ms, Rate without range_ms, Increase, invalid-range_ms).
• +8 tests in engines::simple::engine::sketch_alias_resolver_tests (quantile/count rewrites, no-op cases, topk/rate untouched, identifier-token preservation).
• +4 tests in engines::simple::engine::hll_count_query_tests (count returns cardinality, empty-HLL is 0, Cardinality alias, capability matching).
• +1 test in engines::simple::engine::kll_quantile_query_tests (KLL capability matching).
• +1 test in engines::simple::engine::cms_rate_capability_tests (CMS Rate capability matching, range_ms in kwargs).

Live curl evidence

Backend image rebuilt with this branch; against the mvp-multi-stage stack with a runtime-pushed 5-sketch streaming-config (POST /api/v1/streaming-config adding HLL / KLL / CountSketch / CountMin agg_ids), the previously-error queries now route through capability matching end-to-end (no status=error on capability miss for these query shapes). Live answer values still depend on agent → gateway → backend traffic landing; the producer→agent network in this environment is flaky in this run (DNS resolution failures from producers, surfacing as "no result" rather than "error"). The ingest + capability + query contracts are pinned by the new unit tests.

Pre-fix:

=== count(unique_users_per_min) ===
{"data":null,"error":"No result for query","errorType":"bad_data","infos":["data_source: sketch_warm"],"status":"error"}

Post-fix (with the streaming-config push that registers the HLL agg):

$ curl -sX POST -m 3 'http://localhost:19091/api/v1/streaming-config' -H 'Content-Type: application/x-yaml' --data-binary @streaming-extended.yaml
{"agg_ids_added":[1,5,2,4,3],"agg_ids_removed":[],"new_aggregation_count":5,"schemas_created":[4,3,5],"schemas_retired":[],"status":"success"}

Linked issue

ProjectASAP/ASAPCollector#46 — five-sketch routing pipeline, warm tier serving the five canonical query classes.

🤖 Generated with Claude Code