fix(multinode): harness debt — queries-e2e ranges, b0/b1 query endpoint, aggregate_report

[zzylol]

Summary

Three orthogonal fixes for the multinode harness, surfaced by a clean post-#395 all-arms run:

• Fix 1 — queries-e2e.json [1m] → [5m] (3 entries). The agent window is 60s, backend tumbling window 30s, soak 90s — [1m] ranges fired early in the soak frequently land between open windows and return empty. [5m] matches the convention used everywhere else (smoke test, manual probes, PR fix(deploy): wire controller into multinode harness + replay typo + wave queries + stale bucket key #394 wave queries). quantile by (zone) (...) left alone — no range arg.
• Fix 2 — b0/b1 query endpoint: point arm_measure at VictoriaMetrics :8428, not Prometheus :9090. The harness brings up asap-victoriametrics (not Prometheus) on node2 for b0/b1 (see backend_up), and the agents PRW to victoriametrics:8428/api/v1/write. :9090 was unreachable → 100% of 899 attempts failed with http_code: null (urllib URLError → status: timeout).
• Fix 3 — deploy/mvp-multinode/scripts/aggregate_report.py (new). Fills the missing script run_demo.sh::all referenced. Pure stdlib; handles the multinode dir shape (per-arm subdir with replay.jsonl + edge-<node>.csv + stages-<node>.csv). Emits a summary table, per-query success/empty/error breakdown, edge bandwidth and stages aggregates. Idempotent; failure stays non-fatal.

Fix 2 — root cause + evidence

Live probe of a fresh b0 arm:

$ curl -sm 5 http://10.10.1.3:9090/-/healthy           # exit 7 (connection refused)
$ curl -sm 5 http://10.10.1.3:8428/health              # "OK"
$ ssh node2 'docker ps --format ...'
  asap-victoriametrics: Up ... (image=victoriametrics/victoria-metrics:v1.110.0)
$ curl -sm 5 -G http://10.10.1.3:8428/api/v1/query --data-urlencode 'query=count(http_requests_total)'
  {"status":"success","data":{"result":[{"value":[...,"10000"]}]}, ...}
$ curl -sm 5 -G http://10.10.1.3:8428/api/v1/query --data-urlencode 'query=sum by (zone) (rate(http_requests_total[5m]))'
  {"status":"success","data":{"result":[
    {"metric":{"zone":"z0"},"value":[...,"249990.58..."]},
    {"metric":{"zone":"z1"},"value":[...,"249990.57..."]},
    {"metric":{"zone":"z2"},"value":[...,"249990.06..."]},
    {"metric":{"zone":"z3"},"value":[...,"249989.40..."]}]}}

Quick 15s replay against :8428 returned 78/138 success — the 60 empties are the 3 *_latency_ms queries (no such metric in the b0 workload).

Validation — full all-arms run after fixes

/mydata/mvp-multinode/results/mvp-multinode-20260518-113420/MVP_REPORT.md:

Arm	Total	Success	Empty	Error
asap	413	332	0	81
b0	829	472	357	0
b1	835	476	359	0

Before this PR (same harness, same workload): asap 354/830, b0 0/899, b1 0/899.

asap arm — wave queries from #291/#292/#394 unchanged (no regression):

Query	Total	Success
sum by (zone) (http_requests_total)	59	59
sum by (zone) (rate(http_requests_total[5m]))	59	59
topk(5, sum by (zone) (rate(http_requests_total[5m])))	59	59
sum_over_time(http_requests_total[5m])	59	59

asap arm — [5m]-changed queries:

Query	Total	Success	Was ([1m])
quantile_over_time(0.99, http_requests_total_latency_ms[5m])	59	48	0/119
quantile_over_time(0.5, http_requests_total_latency_ms[5m])	59	48	0/119
sum_over_time(http_requests_total[5m])	59	59	0/119

The remaining error/200=81 on asap are 59 × quantile by (zone) (...) (unsupported PromQL syntax against asap-query-backend, pre-existing) + 11+11 quantile_over_time calls that landed in the first 30s before the first window closed.

b0/b1 arms — wave queries:

Query	b0 success/total	b1 success/total
sum by (zone) (http_requests_total)	118/118	119/119
sum by (zone) (rate(http_requests_total[5m]))	118/118	119/119
sum_over_time(http_requests_total[5m])	118/118	119/119
topk(5, sum by (zone) (rate(http_requests_total[5m])))	118/118	119/119

The 357/359 "empty" results on b0/b1 are the 3 *_latency_ms queries — the workload only emits http_requests_total, so latency_ms returns [] (success / 200 / empty vector, not an error). MVP_REPORT.md correctly classifies these as empty not error.

Test plan

• All-arms run completes end-to-end without error
• asap arm wave queries 59/59 (no fix(gorillas3processor): single critical section for ensureRoleState + consume/flush (TOCTOU race) #395 regression)
• asap arm [5m]-changed quantile_over_time 48/59 (was 0/119)
• b0 arm success > 0 (472/829)
• b1 arm success > 0 (476/835)
• MVP_REPORT.md emitted with per-arm tables
• aggregate_report.py is pure stdlib + idempotent

🤖 Generated with Claude Code