Skip to content

fix(deploy): remove asap-gateway (no longer needed post-wave; saves ~10 Mbps double-hop) - #401

Merged
zzylol merged 1 commit into
mainfrom
fix/deploy-remove-asap-gateway
May 19, 2026
Merged

zzylol merged 1 commit into
mainfrom
fix/deploy-remove-asap-gateway

Conversation

@zzylol

@zzylol zzylol commented May 19, 2026

Copy link
Copy Markdown
Contributor

Rationale

The asap-gateway OTel-collector hop between agents and the asap-backend was originally the cross-agent fan-in tier for sum by (zone) (...) aggregations. After ASAPQuery-backend PRs #283 (B2 stage-emit) and #287/#290 (cumulative emit), the wave-tier backend does that fan-in natively via its evaluate_exact_agg reducer (pulls sids from every agent at query time and merges across hosts).

The gateway became a pure OTLP forwarder that doubled every wire byte (agent → gateway → backend) for no algorithmic reason, costing ~10 Mbps per arm on the multinode demo. See issue #400.

Scope

  • deploy/mvp-multinode/scripts/run_demo.sh — drop gateway_up / gateway_down; remove the gateway from arm_up / arm_down. stop_node node1 stays in teardown so pre-asap pipeline uses ~4.3× more bandwidth than baseline (multi-family fan-out + double-hop) #400 leftover containers get reaped.
  • deploy/mvp-multinode/topology.env — drop the --add-host=gateway: alias.
  • deploy/mvp-multinode/configs/asap/asap-otel-agent-b6-asap-single-sketch.yaml + singlenode equivalent — OTLP endpoint: flips from gateway:4317 to backend:4317.
  • deploy/mvp-{multi,single}node/configs/.../mvp-workload.yaml — the lone assign_to_role: gateway entry flips to assign_to_role: agent. With no Gateway stage in the DAG, the controller's typed split_typed_three_stage simply doesn't emit a Gateway stage and push_to_role(Gateway, ...) is a no-op. No control_plane code change needed.
  • deploy/mvp-multinode/configs/asap/asap-otel-gateway-mvp-placeholder.yaml — deleted.
  • deploy/mvp-singlenode/docker-compose/base.yml — gateway service removed; host ports 14317 / 14318 reassigned to backend. fake-exporter EXPORTER_TARGET defaults to backend:4317; depends_on updated.
  • deploy/mvp-singlenode/docker-compose/agents-N{1,10,100}.yml + gen-agents.sh — depends_on: gateway → depends_on: backend.

Validation (multinode, asap arm, WARMUP_S=90)

docker ps after run_demo.sh up asap:

node containers
node0 agent-a + 5 producers
node1 (empty — no asap-gateway)
node2 backend + controller + minio + prometheus + thanos-{query,store-gateway,compact}
node3 agent-b + 5 producers

All 5 wave queries against http://10.10.1.3:9091/api/v1/query returned data_source: asap_query:

query n_series sample value
quantile_over_time(0.99, http_requests_total_latency_ms[5m]) 10000 per-series p99
max by (zone) (quantile_over_time(0.99, http_requests_total_latency_ms[5m])) 4 z0 = 165.70 ms
sum by (zone) (http_requests_total) 4 z0 = 14888877
sum by (zone) (rate(http_requests_total[5m])) 4 z0 = 159509.04
topk(5, sum by (zone) (rate(http_requests_total[5m]))) 4 z1 = 199208.09

Bandwidth before / after

NIC: enp130s0f0, 60 s sample during steady-state asap arm.

node role RX (Mbps) TX (Mbps)
node0 agent-a → backend 0.16 6.80
node1 (unused; was gateway) 0.00 0.00
node2 backend (receives both agents) 13.56 0.33
node3 agent-b → backend 0.16 6.76

Pre-fix the node1 link was carrying ~13 Mbps in + ~13 Mbps out as a pure forwarder (the double-hop the gateway added). Post-fix that ~26 Mbps of node1 wire traffic is gone; the only cross-host bytes are agent→backend at 13.56 Mbps total.

Test plan

  • run_demo.sh sync succeeds
  • run_demo.sh up asap succeeds with no gateway bring-up
  • node1 docker ps shows no asap-gateway
  • node2 docker ps shows the expected backend stack with no gateway
  • All 5 wave queries return non-empty results via data_source: asap_query
  • run_demo.sh down cleanly tears down all 4 nodes
  • Steady-state aggregate bandwidth ≤ 14 Mbps (was ~27 Mbps with the gateway double-hop)

🤖 Generated with Claude Code

…10 Mbps double-hop)

The asap-gateway OTel-collector hop was originally the cross-agent
fan-in tier for sum-by-zone aggregations. Post ASAPQuery-backend
PRs #283 (B2 stage-emit) and #287/#290 (cumulative emit), the
wave-tier backend does that fan-in natively via the
`evaluate_exact_agg` reducer pulling sids from every agent. The
gateway became a pure forwarder that doubled every byte (agent →
gateway → backend) for no algorithmic reason, costing ~10 Mbps per
arm on the multinode demo (issue #400).

Scope:
- multinode `run_demo.sh`: remove `gateway_up` / `gateway_down`, drop
  the gateway out of `arm_up` / `arm_down`. `stop_node node1` stays in
  the teardown to reap any pre-#400 leftover container; `node1` is
  otherwise unused.
- multinode `topology.env`: drop the `--add-host=gateway:` alias and
  retire the node1 role note.
- multinode + singlenode agent yaml: OTLP `endpoint:` flips from
  `gateway:4317` to `backend:4317`. The backend already accepts OTLP on
  4317 (`--otel-grpc-port=4317` in `run_demo.sh` and base.yml).
- multinode + singlenode `mvp-workload.yaml`: the lone
  `assign_to_role: gateway` entry (sum-by-zone) becomes
  `assign_to_role: agent`. The controller's typed `split_typed_three_stage`
  emits only Edge + Backend stages when no Gateway stage is occupied,
  so `emit_gateway_yaml` simply doesn't fire and `push_to_role(Gateway, ...)`
  is a no-op — no controller code change needed.
- multinode `asap-otel-gateway-mvp-placeholder.yaml`: deleted.
- singlenode `base.yml`: gateway service removed; its host ports
  (14317 / 14318) reassigned to the backend container so external
  tooling that previously hit the gateway can hit the backend on the
  same host ports. `fake-exporter` default `EXPORTER_TARGET` flips to
  `backend:4317`; `depends_on` flips to `backend`.
- singlenode `agents-N{1,10,100}.yml` + `gen-agents.sh`: `depends_on`
  flips from `gateway` → `backend`.

Validation (multinode, 60 s NIC sample, asap arm, WARMUP_S=90):
- node1 docker ps: empty (no asap-gateway).
- node2 docker ps: backend + thanos-{query,store-gateway,compact} +
  minio + prometheus + controller; no gateway.
- All 5 wave queries against node2:9091 return `data_source: asap_query`:
    quantile_over_time(0.99, http_requests_total_latency_ms[5m])
      → 10000 series
    max by (zone) (quantile_over_time(0.99, ...))
      → 4 series
    sum by (zone) (http_requests_total)
      → 4 series
    sum by (zone) (rate(http_requests_total[5m]))
      → 4 series
    topk(5, sum by (zone) (rate(...)))
      → 4 series
- Per-node NIC bandwidth (enp130s0f0, 60 s):
    node0  RX 0.16 Mbps  TX 6.80 Mbps   (agent-a → backend)
    node1  RX 0.00 Mbps  TX 0.00 Mbps   (idle — pre-fix carried the gateway double-hop)
    node2  RX 13.56 Mbps TX 0.33 Mbps   (backend receives both agents)
    node3  RX 0.16 Mbps  TX 6.76 Mbps   (agent-b → backend)
  Pre-fix the node1 link carried ~13 Mbps in + ~13 Mbps out as a pure
  forwarder. Post-fix that ~26 Mbps of node1 wire traffic is gone; the
  remaining ~13.6 Mbps lives only on the agent→backend hop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit eabd1bf into main May 19, 2026
@zzylol
zzylol deleted the fix/deploy-remove-asap-gateway branch July 17, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant