asap engine: sum by (...) (http_requests_total) returns ~10x the baseline value (Sum semantic divergence)

[zzylol]

Symptom

sum by (zone) (http_requests_total) on asap tier returns dramatically larger values than the VictoriaMetrics baseline for the same instant query against the same producer fleet.

Discovered while computing apples-to-apples accuracy on multinode demo (post ASAPCollector PR fixing per-series DDSketch grouping; quantile queries now show 0-12% rel-err which IS sketch ε behavior). Sum/Rate/Topk values diverge by 3 orders of magnitude.

Observed (multinode all-arms, run dir mvp-multinode-20260518-150530)

Query	asap mean	b0 mean (VictoriaMetrics raw)	Rel-err
sum by (zone) (http_requests_total)	777,183,514	20,725,053	3650%
sum by (zone) (rate(http_requests_total[5m]))	3,180,661	249,994	1172%
topk(5, sum by (zone) (rate(...)))	3,180,661	249,994	1172%
sum_over_time(http_requests_total[5m])	3,108,734,057	381,065	815,701%

For context — quantile queries on the SAME run (apples-to-apples after the per-series sketch fix):

Query	asap mean	b0 mean	Rel-err
quantile_over_time(0.5, http_requests_total_latency_ms[5m])	20.153	20.153	0.004%
quantile_over_time(0.99, http_requests_total_latency_ms[5m])	81.365	92.235	11.785%

So the quantile accuracy story works correctly; the divergence is specific to Sum-typed reads.

Reproduction

Post ASAPCollector PR #397 + #298 + post-this-investigation, the multinode harness drops grouping_labels on the latency metric. http_requests_total (Counter) keeps grouping_labels: [zone] for sketch-cost reasons.

cd /mydata/ASAPCollector
bash deploy/mvp-multinode/scripts/run_demo.sh arm asap
# wait 90s for first window flush
curl -sG "http://10.10.1.3:9091/api/v1/query" \
     --data-urlencode 'query=sum by (zone) (http_requests_total)'
# → returns per-zone values orders-of-magnitude larger than the producer's actual counter

Producer emits at 10 Hz × 25 series per zone for ~150s = ~37,500 events per zone. VictoriaMetrics returns ~20M per zone (counter value ~= cumulative events). Asap returns ~194M per zone — ~10× larger than the actual counter.

Hypotheses to investigate

1. Lifetime-accumulated Sum semantics — the asap engine's ExactAgg(Sum) read may be summing ALL per-window Sums across the sketch's full history, not returning the most recent window's value (or the equivalent of "current counter").

2. Multi-path ingest amplification — the asap pipeline has raw_passthrough + ExactAgg(Sum) sketch landing at the backend. If both paths register Sum aggregates for the same metric and the engine sums both, you'd see ~2× per event. Doesn't match the ~10× ratio cleanly though.

3. Gateway-level Sum accumulation — agent → gateway → backend. If the gateway aggregates Sum-per-window and the backend also aggregates the gateway's pre-aggregated output, you'd see windowed amplification.

4. Per-series-then-per-zone double-rollup — the sketch is keyed by (metric, [zone]) grouping but the source emits per-series, and the agent's attributes/keep might not be dropping the rolled-up labels before sketching → multiple sketches per zone get merged into one but with overlapping samples.

5. PromQL sum() semantic on ExactAgg(Sum) sids — VictoriaMetrics' sum by (zone) (counter) returns sum-of-current-counter-values. Asap's engine for ExactAgg(Sum) may return sum-of-window-sums (e.g., events-per-window summed across the storage horizon), which would be a different quantity entirely from PromQL's sum(counter) semantic.

Suggested investigation steps

1. Probe /api/v1/db/schemas during the asap arm + dump per-sid sample counts and Sum values. Compare to "expected events per sid".
2. Compare instant vs range read paths: asap's query_range (matrix shape) vs query (instant). Does query_range show counter-like growth (expected for VictoriaMetrics-style Sum read) or step-function-windowed-sums (asap's apparent semantic)?
3. Check the ExactAgg(Sum) reducer's instant-query semantic. PromQL sum(counter) at instant t should return the latest counter value. If the reducer instead returns Σ-over-windows-of-window-Sum, that's the divergence.
4. Trace raw_passthrough vs sketch sids for http_requests_total. If the metric has multiple registered sids (raw + sketched + ...) and the engine sums all of them, the magnitude would multiply.

Impact

Low for single-metric correctness validation — quantile path works. Medium for the multinode demo report — Sum/Rate/Topk columns show meaningless rel-errs against VictoriaMetrics, undermining the demo's accuracy story for those query classes. High for any production deployment that intends to use asap as a counter store.

Reference

• Multinode run dir with both arms' raw replay data: /mydata/mvp-multinode-150530/{asap,b0,b1}/replay.jsonl
• Accuracy comparison script (stdlib Python): /tmp/accuracy.py (mean-across-soak per query, asap vs baselines)
• Quantile accuracy validation (proves the harness setup is correct elsewhere): same run, latency queries 0-12% rel-err