fix(deploy): remove asap-gateway (no longer needed post-wave; saves ~10 Mbps double-hop) - #401
Merged
Merged
Conversation
…10 Mbps double-hop) The asap-gateway OTel-collector hop was originally the cross-agent fan-in tier for sum-by-zone aggregations. Post ASAPQuery-backend PRs #283 (B2 stage-emit) and #287/#290 (cumulative emit), the wave-tier backend does that fan-in natively via the `evaluate_exact_agg` reducer pulling sids from every agent. The gateway became a pure forwarder that doubled every byte (agent → gateway → backend) for no algorithmic reason, costing ~10 Mbps per arm on the multinode demo (issue #400). Scope: - multinode `run_demo.sh`: remove `gateway_up` / `gateway_down`, drop the gateway out of `arm_up` / `arm_down`. `stop_node node1` stays in the teardown to reap any pre-#400 leftover container; `node1` is otherwise unused. - multinode `topology.env`: drop the `--add-host=gateway:` alias and retire the node1 role note. - multinode + singlenode agent yaml: OTLP `endpoint:` flips from `gateway:4317` to `backend:4317`. The backend already accepts OTLP on 4317 (`--otel-grpc-port=4317` in `run_demo.sh` and base.yml). - multinode + singlenode `mvp-workload.yaml`: the lone `assign_to_role: gateway` entry (sum-by-zone) becomes `assign_to_role: agent`. The controller's typed `split_typed_three_stage` emits only Edge + Backend stages when no Gateway stage is occupied, so `emit_gateway_yaml` simply doesn't fire and `push_to_role(Gateway, ...)` is a no-op — no controller code change needed. - multinode `asap-otel-gateway-mvp-placeholder.yaml`: deleted. - singlenode `base.yml`: gateway service removed; its host ports (14317 / 14318) reassigned to the backend container so external tooling that previously hit the gateway can hit the backend on the same host ports. `fake-exporter` default `EXPORTER_TARGET` flips to `backend:4317`; `depends_on` flips to `backend`. - singlenode `agents-N{1,10,100}.yml` + `gen-agents.sh`: `depends_on` flips from `gateway` → `backend`. Validation (multinode, 60 s NIC sample, asap arm, WARMUP_S=90): - node1 docker ps: empty (no asap-gateway). - node2 docker ps: backend + thanos-{query,store-gateway,compact} + minio + prometheus + controller; no gateway. - All 5 wave queries against node2:9091 return `data_source: asap_query`: quantile_over_time(0.99, http_requests_total_latency_ms[5m]) → 10000 series max by (zone) (quantile_over_time(0.99, ...)) → 4 series sum by (zone) (http_requests_total) → 4 series sum by (zone) (rate(http_requests_total[5m])) → 4 series topk(5, sum by (zone) (rate(...))) → 4 series - Per-node NIC bandwidth (enp130s0f0, 60 s): node0 RX 0.16 Mbps TX 6.80 Mbps (agent-a → backend) node1 RX 0.00 Mbps TX 0.00 Mbps (idle — pre-fix carried the gateway double-hop) node2 RX 13.56 Mbps TX 0.33 Mbps (backend receives both agents) node3 RX 0.16 Mbps TX 6.76 Mbps (agent-b → backend) Pre-fix the node1 link carried ~13 Mbps in + ~13 Mbps out as a pure forwarder. Post-fix that ~26 Mbps of node1 wire traffic is gone; the remaining ~13.6 Mbps lives only on the agent→backend hop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale
The asap-gateway OTel-collector hop between agents and the asap-backend was originally the cross-agent fan-in tier for
sum by (zone) (...)aggregations. After ASAPQuery-backend PRs #283 (B2 stage-emit) and #287/#290 (cumulative emit), the wave-tier backend does that fan-in natively via itsevaluate_exact_aggreducer (pulls sids from every agent at query time and merges across hosts).The gateway became a pure OTLP forwarder that doubled every wire byte (agent → gateway → backend) for no algorithmic reason, costing ~10 Mbps per arm on the multinode demo. See issue #400.
Scope
deploy/mvp-multinode/scripts/run_demo.sh— dropgateway_up/gateway_down; remove the gateway fromarm_up/arm_down.stop_node node1stays in teardown so pre-asap pipeline uses ~4.3× more bandwidth than baseline (multi-family fan-out + double-hop) #400 leftover containers get reaped.deploy/mvp-multinode/topology.env— drop the--add-host=gateway:alias.deploy/mvp-multinode/configs/asap/asap-otel-agent-b6-asap-single-sketch.yaml+ singlenode equivalent — OTLPendpoint:flips fromgateway:4317tobackend:4317.deploy/mvp-{multi,single}node/configs/.../mvp-workload.yaml— the loneassign_to_role: gatewayentry flips toassign_to_role: agent. With no Gateway stage in the DAG, the controller's typedsplit_typed_three_stagesimply doesn't emit a Gateway stage andpush_to_role(Gateway, ...)is a no-op. No control_plane code change needed.deploy/mvp-multinode/configs/asap/asap-otel-gateway-mvp-placeholder.yaml— deleted.deploy/mvp-singlenode/docker-compose/base.yml— gateway service removed; host ports 14317 / 14318 reassigned to backend. fake-exporterEXPORTER_TARGETdefaults tobackend:4317;depends_onupdated.deploy/mvp-singlenode/docker-compose/agents-N{1,10,100}.yml+gen-agents.sh—depends_on: gateway→depends_on: backend.Validation (multinode, asap arm, WARMUP_S=90)
docker psafterrun_demo.sh up asap:All 5 wave queries against
http://10.10.1.3:9091/api/v1/queryreturneddata_source: asap_query:quantile_over_time(0.99, http_requests_total_latency_ms[5m])max by (zone) (quantile_over_time(0.99, http_requests_total_latency_ms[5m]))sum by (zone) (http_requests_total)sum by (zone) (rate(http_requests_total[5m]))topk(5, sum by (zone) (rate(http_requests_total[5m])))Bandwidth before / after
NIC:
enp130s0f0, 60 s sample during steady-state asap arm.Pre-fix the node1 link was carrying ~13 Mbps in + ~13 Mbps out as a pure forwarder (the double-hop the gateway added). Post-fix that ~26 Mbps of node1 wire traffic is gone; the only cross-host bytes are agent→backend at 13.56 Mbps total.
Test plan
run_demo.sh syncsucceedsrun_demo.sh up asapsucceeds with no gateway bring-upnode1 docker psshows noasap-gatewaynode2 docker psshows the expected backend stack with no gatewaydata_source: asap_queryrun_demo.sh downcleanly tears down all 4 nodes🤖 Generated with Claude Code