asap engine: sum_over_time(counter[r]) still inflated ~14000% post-#299 (semantic gap between per-window-delta-sum and baseline scrape-sum)

[zzylol]

Symptom

After #298 / #299 (the cumulativetodelta emit fix), the marquee sum by (zone) (http_requests_total) accuracy is much better — went from 3650% off baseline to 33.5% (the residual ~33% is the design-intent semantic difference: asap returns per-window-delta vs baseline's cumulative-since-counter-start).

But sum_over_time(http_requests_total[5m]) is still ~14,284% off:

Query	Baseline mean	Asap mean	Rel-err
sum by (zone) (http_requests_total)	20,753,853	13,790,210	33.6% ✅
sum by (zone) (rate(http_requests_total[5m]))	249,993	67,794	72.9%
sum_over_time(http_requests_total[5m])	382,182	54,974,081	14,284% ❌

So rate(counter[5m]) is reasonably close (evaluate_exact_agg_rate handles delta windows correctly), but sum_over_time(counter[r]) is wildly off.

Suspected root cause

sum_over_time(X[r]) in PromQL semantics: for each series, return Σ-over-samples-in-window of X.

• Baseline (VictoriaMetrics): X is cumulative counter, sampled at scrape rate (~1Hz). Over 5min that's ~300 samples. Sum = Σ-cumulative-at-each-scrape ≈ (mean × 300). For a series with current cumulative value V at end of window, mean ≈ V/2 (linear ramp from 0). So sum_over_time ≈ V × 150 per series → reasonable ballpark.
• Asap (post-fix(emit): inject cumulativetodelta upstream of agent routing for Counter metrics (closes #298) #299): per-window deltas. SumAccumulator at backend sums deltas into per-window totals. The reducer's evaluate_exact_agg for sum_over_time returns... what? Likely Σ-of-per-window-deltas-across-the-[5m]-range, multiplied by something (sample-count? window-count?). The 14,000% inflation suggests there's still a quadratic-ish accumulation somewhere — possibly the engine's evaluate_exact_agg instant-Sum path is summing across all stored windows (not just the [r]-bounded ones), or there's a per-sample-count multiplier still in effect for the sum_over_time function specifically.

Diagnostic next steps

1. Probe sum_over_time(http_requests_total[30s]) (single window) vs sum_over_time(http_requests_total[5m]) (10 windows). If 5m is exactly 10× the 30s result, the engine is correctly summing per-window-delta over the range — and the 14000% gap is the semantic difference between "sum of N cumulative samples" (baseline) and "sum of M per-window deltas" (asap), which is fundamentally different by ratio ~N×V/2 / M×V.
2. Read evaluate_exact_agg dispatch for sum_over_time. Is the function name routed to a per-window iter + sum, or to something else? Is there a multiplier from the function_args (range_seconds)?
3. Compare sum_over_time(counter[r]) to sum_over_time(gauge[r]). Gauges don't have temporality, so if both are inflated, the bug is in the engine's sum_over_time dispatch. If only counters are inflated, the bug is something in the cumulativetodelta → SumAccumulator → sum_over_time chain.

Hypotheses

1. sum_over_time is treating window-deltas as if they were each cumulative-counter samples — summing them gives Σ-of-deltas = total events in range, but baseline expects Σ-of-cumulative-samples = quadratic-in-rate. These are intrinsically different quantities.

2. Engine's per-window-fold for sum_over_time has a residual cumulative-vs-delta mismatch — maybe the agent's cumulativetodelta processor doesn't apply to whatever path sum_over_time reads from, or the reducer does something funky.

3. Honest acknowledgement might be: "asap's sum_over_time(counter) semantic is fundamentally different from baseline's because asap stores per-window deltas" — in which case the right fix is to NOT support this PromQL idiom on asap (CapabilityMiss → archive), since the math literally can't match.

Impact

Low — sum_over_time(counter) is an uncommon PromQL idiom (rate is preferred for counter analysis). But the demo's accuracy report shows a glaring 14,000% rel-err column, which undermines the demo narrative. Worth a tight follow-up after the multinode demo's other validation items.

Reference

• Pre-fix: 815,701% off (cubic blowup)
• Post-fix(emit): inject cumulativetodelta upstream of agent routing for Counter metrics (closes #298) #299: 14,284% off (quadratic-ish remainder)
• Multinode run dir: /mydata/mvp-multinode/results/mvp-multinode-20260518-182343/
• Accuracy script: /tmp/accuracy.py