Conversation
Top-level TODO, deploy/README, deploy/TODO all predated the multi-agent scaffold (#168–#185) and no longer matched the code. Realities the docs now reflect: - deploy scaffold (compose base + N∈{1,10,100} overlays, 7 baselines, 5 Dockerfiles, Helm values, sweep driver) is in. - PR #185 surfaced a system-wide throughput collapse at N=10 (all baselines throttled to ~2k pts/s, agents near-idle) — now called out as the P0 paper blocker instead of "build multi-agent scaffold". - Sweep CSV has nan cells for bandwidth / gateway / backend metrics on some baselines, and backend_query_p99_ms is nan everywhere (no query-side driver yet) — captured as the P1 instrumentation gap. - Grafana dashboards and Helm templates are still unwritten — surfaced honestly rather than hidden under "done". No code changes, docs only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #185 flagged a universal ~2k pts/s floor at N=10 across all six baselines and attributed it to a coordinated throttle (OTLP SDK / Docker userland-proxy / kernel socket buffers). Rate-invariance test disproves the bottleneck reading. Holding cardinality=1000 and varying EXPORTER_RATE from 1000 to 10000 (10× change) on the same b0a-raw-stream N=1 stack: gateway rate stays flat at 2,000 pts/s and backend rate at 2,001 pts/s. If any of the candidate bottlenecks were the cause, 10× input would produce observable throughput delta — it doesn't. Root cause: PR #182 (`feat(workload): fake-exporter trace-replay mode`) changed the OTel MeterProvider's PeriodicReader interval from `time.Second / time.Duration(rate)` to `time.Second` fixed. With a 1 s interval the SDK pre-aggregates Counter.Add and Gauge.Record calls per attribute set within each tick, so the export rate becomes `cardinality × #instruments × (1/interval)` = 1000 × 2 × 1 = 2000 pts/s, independent of input rate. N=10 is just 10 concurrent producers each correctly emitting 2k. Bigger implication: the paper §6.2 "raw vs sketch bandwidth" story is more fragile than it looks. With the current fake-exporter, the "raw" baselines (B0a / B0b / B1) are already SDK pre-aggregated at the producer — they are not emitting per-sample traffic. The bandwidth delta vs sketch baselines measures payload shape, not "raw samples vs summary per window". Lays out four paper-story options (A revert interval / B dual- interval / C bypass SDK aggregation / D reframe §6.2 as bytes- per-window). Recommends B + D. Does not land a fix; that's a follow-up PR that needs a paper-framing decision. Artifacts: - docs/n10-bottleneck-rca.md — full write-up - deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv — the two-row evidence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #185 flagged a universal ~2k pts/s floor at N=10 across all six baselines and attributed it to a coordinated throttle (OTLP SDK / Docker userland-proxy / kernel socket buffers). This branch presents evidence it's none of those — the cause is in our own fake-exporter — and lays out the paper-framing decision that follows.
Rate-invariance evidence (
deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv): holding cardinality=1000 and varyingEXPORTER_RATEfrom 1000 to 10000 (10× change) on the sameb0a-raw-streamN=1 stack, gateway rate stays flat at 2,000 pts/s and backend rate at 2,001 pts/s. If any of the candidate bottlenecks were real, 10× input would shift throughput. It doesn't.Root cause: PR #182 (
feat(workload): fake-exporter trace-replay mode) changed the OTelMeterProvider'sPeriodicReaderinterval fromtime.Second / time.Duration(rate)to a fixedtime.Second. With a 1 s interval the SDK pre-aggregatesCounter.AddandGauge.Recordcalls per attribute set per tick, so export rate becomescardinality × #instruments × (1/interval)= 1000 × 2 × 1 = 2000 pts/s, independent of input rate. N=10 is just 10 producers each correctly emitting 2k.Bigger implication for paper §6.2: the "raw vs sketch bandwidth" story is more fragile than it looks. With the current fake-exporter, the "raw" baselines (B0a / B0b / B1) are already SDK-pre-aggregated at the producer — they are not emitting per-sample traffic. The bandwidth delta vs sketch baselines measures payload shape, not "raw samples vs summary per window."
Paper options laid out in the doc: (A) revert interval, (B) dual-interval mode, (C) bypass SDK aggregation, (D) reframe §6.2 as bytes-per-window. Recommendation: B + D.
What this PR does
docs/n10-bottleneck-rca.md— full write-up (229 lines)deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv— the two-row evidenceTODO.md/deploy/README.md/deploy/TODO.md— sync to post-sweep reality: scaffold landed, P0 is the throughput-floor finding, P1 is the nan instrumentation gap, dashboards/Helm templates surfaced honestly as unwritten.What this PR does NOT do
No code change to the fake-exporter. The fix needs a paper-framing decision first (which of A/B/C/D), and that's a separate PR.
Test plan
🤖 Generated with Claude Code