asap engine: outer aggregation operators on function results aren't composed

[zzylol]

Symptom

max by (zone) (quantile_over_time(0.99, http_requests_total_latency_ms[5m])) returns {"error":"No result for query","errorType":"bad_data","infos":["data_source: asap_query"],"status":"error"} on the asap tier, even though the inner quantile_over_time(...) succeeds and returns per-zone p99 values. The same PromQL succeeds 100% (118/118) on the b0/b1 VictoriaMetrics baselines.

Discovered during multinode validation post ASAPCollector PR #397 (the harness debt closeout). All 6 other wave queries are 100% success on multinode asap arm — this is the one outlier:

Query	asap	b0	b1
quantile_over_time(0.99, ...[5m])	54/54 ✅	119/119	118/118
quantile_over_time(0.5, ...[5m])	53/53 ✅	119/119	118/118
sum by (zone) (http_requests_total)	53/53 ✅	118/118	118/118
sum by (zone) (rate(...))	53/53 ✅	118/118	118/118
sum_over_time(...[5m])	53/53 ✅	118/118	118/118
topk(5, sum by (zone) (rate(...)))	53/53 ✅	118/118	118/118
max by (zone) (quantile_over_time(...))	0/53 ❌	118/118	118/118

Reproduction

Against any asap-backend with a DDSketch sketch for http_requests_total_latency_ms (e.g., post ASAPCollector PR #397 multinode asap arm or the singlenode smoke harness):

# Inner works:
curl -sm 10 -G "http://10.10.1.3:9091/api/v1/query" \
     --data-urlencode 'query=quantile_over_time(0.99, http_requests_total_latency_ms[5m])'
# → {"data":{"result":[{...},{...},{...},{...}], ...},"status":"success"}  (4 zones)

# Wrapping in an aggregation operator fails:
curl -sm 10 -G "http://10.10.1.3:9091/api/v1/query" \
     --data-urlencode 'query=max by (zone) (quantile_over_time(0.99, http_requests_total_latency_ms[5m]))'
# → {"error":"No result for query","errorType":"bad_data","infos":["data_source: asap_query"],"status":"error"}

Note: the inner already returns per-zone results (the sketch is grouped by [zone] per the workload's grouping_labels). So the outer max by (zone) is semantically a no-op identity — it should just return the same 4 zones.

Likely root cause

The engine's PromQL evaluator (data_plane/src/query_engines/asap_query_engine/engine.rs) has typed dispatch for various function shapes (post-#291, #292, #295), but the outer aggregation operator (max, min, avg, stddev, stdvar, group, etc.) wrapping a function result isn't recognized. The query likely fails at capability matching or sid resolution because the analyzer / matcher doesn't know what to do with the aggregation-of-function shape.

For reference, the engine DOES handle:

• sum by (...) (Sum-aggregation metric) — feat(query): dispatch sum by (...) PromQL to ExactAgg(Sum) sids (closes Option-B downstream) #291 evaluate_exact_agg
• sum by (...) (rate(...)) and topk(K, sum by (...) (rate(...))) — feat(query): compose rate-over-Sum into ExactAgg(Sum) candidate (multinode topk dispatch) #292 rate-over-Sum compose + topk lift
• quantile_over_time(0.99, ...) — DDSketch reducer (when sketch is grouped to query's by-clause)

But NOT:

• max by (...) (quantile_over_time(...))
• avg by (...) (quantile_over_time(...))
• probably also min by, stddev by, etc. on function results

Suggested fix shape

The cleanest is probably to extend the analyzer's ASAPTierCandidate (or equivalent — see how #295 added OuterFn) so outer aggregation operators are typed-tracked alongside the inner function. The engine then evaluates:

1. Inner function → list of (labels, value) rows
2. Apply outer aggregation operator using the standard PromQL semantics (per-group fold based on the operator + by-clause)

The no-op identity case (when inner is ALREADY grouped to the same by-clause as the outer) should fall out naturally — fold each group of 1 with max returns the same value.

Doable mostly in data_plane/src/query_engines/asap_query_engine/. Probably ~200-400 LOC including tests.

Impact

Low priority — only affects ad-hoc PromQL queries that wrap function results in aggregation operators. The wave's primary use cases (sum by, rate, topk, quantile_over_time) all work. But this gap surfaces in cross-tier accuracy comparisons (baselines can answer, asap can't), so it's a real demo-quality issue for any benchmark report that includes such queries.

Tagged bug (the demo's MVP_REPORT.md surfaces a 0/53 on asap for this query while baselines are 118/118).