eval(N=10): scale sweep surfaces systemic producer bottleneck - #185
Merged
Merged
Conversation
Full 6-baseline sweep at N=10, same config as the N=1 run
(rate=1000/s, cardinality=1000, SOAK_S=180, 60s windows where
applicable).
Per-agent throughput collapsed to a universal ~2,000 pts/s floor
across EVERY baseline at N=10, vs 130k–326k pts/s at N=1:
baseline N=1 pts/s → N=10 pts/s
b0a-raw-stream 325k → 2k
b0b-raw-batched 268k → 2k
b1-serf 143k → 2k
b2-full 278k → 2k
b3-delta 274k → 2k
b5-gorilla 130k → 2k
Gateway end-to-end throughput for the raw baselines matches
10 × 2k = 20k/s, so all 10 agents are coordinated-throttled to
the same per-agent rate. Agents' CPU and RSS are near-idle
(0.01c, ~220 MiB), ruling out agent-side OOM or CPU saturation.
This is a SYSTEM bottleneck, not a per-baseline effect — B0 raw
forward is affected identically to B3 delta sketch. Probable
causes (untested, ranked by plausibility):
* OTLP/gRPC SDK backpressure in fake-exporter: 10 producers
× 1M points/s nominal (rate × cardinality) overwhelms some
shared socket / SDK reader limit. At N=1 one producer
reaches ~325k effective; at N=10 each reaches 2k — 150×
degradation per producer, 16× aggregate degradation (20k
vs 325k total), consistent with a per-process quota-like
bottleneck in the producer SDK.
* Docker userland-proxy CPU contention across many simultaneous
gRPC connections on the bridge network.
* Kernel socket-buffer limits (net.core.somaxconn /
net.core.rmem_max) pushed past default.
Paper implications: the §6.2 N=10 row in the current matrix
should NOT be read as a scale-quality claim — it's measuring
the producer bottleneck, not the sketch pipeline. Either:
- switch fake-exporter to a non-OTLP-SDK emission path
(raw gRPC client) for the scale sweep, or
- tune OTLP SDK batch / queue limits (WithMaxQueueSize,
WithMaxExportBatchSize) to see whether they account for
the floor, or
- accept N=1 as the paper's load-quality datapoint and treat
N=10 purely as a "stack stays up" scale-stability test.
Archived honestly. Diagnosing the bottleneck is a follow-up PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
Top-level TODO, deploy/README, deploy/TODO all predated the multi-agent scaffold (#168–#185) and no longer matched the code. Realities the docs now reflect: - deploy scaffold (compose base + N∈{1,10,100} overlays, 7 baselines, 5 Dockerfiles, Helm values, sweep driver) is in. - PR #185 surfaced a system-wide throughput collapse at N=10 (all baselines throttled to ~2k pts/s, agents near-idle) — now called out as the P0 paper blocker instead of "build multi-agent scaffold". - Sweep CSV has nan cells for bandwidth / gateway / backend metrics on some baselines, and backend_query_p99_ms is nan everywhere (no query-side driver yet) — captured as the P1 instrumentation gap. - Grafana dashboards and Helm templates are still unwritten — surfaced honestly rather than hidden under "done". No code changes, docs only. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
* docs: RCA — "N=10 throughput collapse" is not a bottleneck PR #185 flagged a universal ~2k pts/s floor at N=10 across all six baselines and attributed it to a coordinated throttle (OTLP SDK / Docker userland-proxy / kernel socket buffers). Rate-invariance test disproves the bottleneck reading. Holding cardinality=1000 and varying EXPORTER_RATE from 1000 to 10000 (10× change) on the same b0a-raw-stream N=1 stack: gateway rate stays flat at 2,000 pts/s and backend rate at 2,001 pts/s. If any of the candidate bottlenecks were the cause, 10× input would produce observable throughput delta — it doesn't. Root cause: PR #182 (`feat(workload): fake-exporter trace-replay mode`) changed the OTel MeterProvider's PeriodicReader interval from `time.Second / time.Duration(rate)` to `time.Second` fixed. With a 1 s interval the SDK pre-aggregates Counter.Add and Gauge.Record calls per attribute set within each tick, so the export rate becomes `cardinality × #instruments × (1/interval)` = 1000 × 2 × 1 = 2000 pts/s, independent of input rate. N=10 is just 10 concurrent producers each correctly emitting 2k. Bigger implication: the paper §6.2 "raw vs sketch bandwidth" story is more fragile than it looks. With the current fake-exporter, the "raw" baselines (B0a / B0b / B1) are already SDK pre-aggregated at the producer — they are not emitting per-sample traffic. The bandwidth delta vs sketch baselines measures payload shape, not "raw samples vs summary per window". Lays out four paper-story options (A revert interval / B dual- interval / C bypass SDK aggregation / D reframe §6.2 as bytes- per-window). Recommends B + D. Does not land a fix; that's a follow-up PR that needs a paper-framing decision. Artifacts: - docs/n10-bottleneck-rca.md — full write-up - deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv — the two-row evidence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: three-axis SDK aggregation framework + sync stale docs Formalizes the SDK-side decision point as a (W, L, agg_type) triple: time window × label projection × encoding. All three axes are independent and correspond one-to-one to what the controller's planner emits per metric. - docs/sdk-aggregation-three-axis-design.md (new) — authoritative design doc for the framework. Maps to existing Mode 1/2/3 vocabulary in delta-transmission-design.md. Enumerates the outstanding SDK-side aggregators (raw-buffer + 5 delta variants) and the SDK runtime hook (hot-reload AttributeFilter) needed to exercise the full planner loop. - docs/paper-outline.md §6 — split §6.2 into four sub-sweeps (a/b/c/d) along the three axes, introduce §6.5 planner-quality as an independent experiment, update claims table to express bandwidth reduction as a three-factor product. - PROGRESS.md — bump date to 2026-04-23; point at new design doc; enumerate outstanding aggregators (~150 LOC raw-buffer + ~500 LOC five delta variants) + fake-exporter knobs + measure-baseline producer-side columns. - docs/n10-bottleneck-rca.md — prepend postscript noting the Options A–D recommendation section is superseded by the three-axis design. Diagnosis content unchanged. - TODO.md (top-level) — retire "N=10 throughput collapse" as P0 blocker (the RCA closed it); replace with the concrete implementation punch list for the three-axis framework. - deploy/TODO.md — add producer-side measurement requirement (producer_cpu_cores / _rss_mib / _bytes_out_per_s) to the instrumentation P1 list; the three-axis sweeps read these from fake-exporter container directly rather than inferring from gateway counters. Includes the n10-bottleneck-rca.md content from the earlier diag/n10-bottleneck-rca branch (superseded by this PR — close that one). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: correct scope — delta-encoding is a flag on existing aggregators Reading opentelemetry-go-patch/sdk/metric/aggregation.go shows that DeltaTransmission is already a field on DDSketch / CountSketch / CountMinSketch / HLLSketch aggregators (landed 2026-03-14), so there's no need to ship five new AggregationDelta<X> types. Only KLL lacks delta support, and its multi-level sample-buffer structure doesn't admit a naive byte-diff — separate design problem, not a §6.2 blocker. Updates docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, and TODO.md to reflect this. Real remaining gap: AggregationRawBuffer (~150 LOC). Proceeding to implement that next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
* docs: RCA — "N=10 throughput collapse" is not a bottleneck PR #185 flagged a universal ~2k pts/s floor at N=10 across all six baselines and attributed it to a coordinated throttle (OTLP SDK / Docker userland-proxy / kernel socket buffers). Rate-invariance test disproves the bottleneck reading. Holding cardinality=1000 and varying EXPORTER_RATE from 1000 to 10000 (10× change) on the same b0a-raw-stream N=1 stack: gateway rate stays flat at 2,000 pts/s and backend rate at 2,001 pts/s. If any of the candidate bottlenecks were the cause, 10× input would produce observable throughput delta — it doesn't. Root cause: PR #182 (`feat(workload): fake-exporter trace-replay mode`) changed the OTel MeterProvider's PeriodicReader interval from `time.Second / time.Duration(rate)` to `time.Second` fixed. With a 1 s interval the SDK pre-aggregates Counter.Add and Gauge.Record calls per attribute set within each tick, so the export rate becomes `cardinality × #instruments × (1/interval)` = 1000 × 2 × 1 = 2000 pts/s, independent of input rate. N=10 is just 10 concurrent producers each correctly emitting 2k. Bigger implication: the paper §6.2 "raw vs sketch bandwidth" story is more fragile than it looks. With the current fake-exporter, the "raw" baselines (B0a / B0b / B1) are already SDK pre-aggregated at the producer — they are not emitting per-sample traffic. The bandwidth delta vs sketch baselines measures payload shape, not "raw samples vs summary per window". Lays out four paper-story options (A revert interval / B dual- interval / C bypass SDK aggregation / D reframe §6.2 as bytes- per-window). Recommends B + D. Does not land a fix; that's a follow-up PR that needs a paper-framing decision. Artifacts: - docs/n10-bottleneck-rca.md — full write-up - deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv — the two-row evidence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: three-axis SDK aggregation framework + sync stale docs Formalizes the SDK-side decision point as a (W, L, agg_type) triple: time window × label projection × encoding. All three axes are independent and correspond one-to-one to what the controller's planner emits per metric. - docs/sdk-aggregation-three-axis-design.md (new) — authoritative design doc for the framework. Maps to existing Mode 1/2/3 vocabulary in delta-transmission-design.md. Enumerates the outstanding SDK-side aggregators (raw-buffer + 5 delta variants) and the SDK runtime hook (hot-reload AttributeFilter) needed to exercise the full planner loop. - docs/paper-outline.md §6 — split §6.2 into four sub-sweeps (a/b/c/d) along the three axes, introduce §6.5 planner-quality as an independent experiment, update claims table to express bandwidth reduction as a three-factor product. - PROGRESS.md — bump date to 2026-04-23; point at new design doc; enumerate outstanding aggregators (~150 LOC raw-buffer + ~500 LOC five delta variants) + fake-exporter knobs + measure-baseline producer-side columns. - docs/n10-bottleneck-rca.md — prepend postscript noting the Options A–D recommendation section is superseded by the three-axis design. Diagnosis content unchanged. - TODO.md (top-level) — retire "N=10 throughput collapse" as P0 blocker (the RCA closed it); replace with the concrete implementation punch list for the three-axis framework. - deploy/TODO.md — add producer-side measurement requirement (producer_cpu_cores / _rss_mib / _bytes_out_per_s) to the instrumentation P1 list; the three-axis sweeps read these from fake-exporter container directly rather than inferring from gateway counters. Includes the n10-bottleneck-rca.md content from the earlier diag/n10-bottleneck-rca branch (superseded by this PR — close that one). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: correct scope — delta-encoding is a flag on existing aggregators Reading opentelemetry-go-patch/sdk/metric/aggregation.go shows that DeltaTransmission is already a field on DDSketch / CountSketch / CountMinSketch / HLLSketch aggregators (landed 2026-03-14), so there's no need to ship five new AggregationDelta<X> types. Only KLL lacks delta support, and its multi-level sample-buffer structure doesn't admit a naive byte-diff — separate design problem, not a §6.2 blocker. Updates docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, and TODO.md to reflect this. Real remaining gap: AggregationRawBuffer (~150 LOC). Proceeding to implement that next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(sdk/metric): AggregationRawBuffer — raw-sample encoding slot Adds AggregationRawBuffer to the SDK as the encoding-axis baseline for the three-axis framework (see docs/sdk-aggregation-three-axis-design.md): where every Counter.Add / Gauge.Record call becomes its own NumberDataPoint on the wire instead of being reduced to a Sum or LastValue per attribute set per tick. Implementation: - sdk/metric/aggregation.go — new AggregationRawBuffer{MaxEventsPerSeries} public type, wired into the Aggregation interface and err() validation alongside the existing sketch aggregators. - sdk/metric/internal/aggregate/rawbuffer.go — aggregator impl. Per attribute.Distinct key, appends (ts, value) to a bounded slice in measure(); in collect() emits each buffered sample as its own metricdata.DataPoint[N] inside a Gauge[N], then clears the buffer. Overflow on a single series is silently dropped with a per-series drop counter (exposing that via a side-channel metric is a follow-up tracked in PROGRESS.md). - sdk/metric/internal/aggregate/aggregate.go — Builder.RawBuffer method, mirroring the KLLSketch / HLLSketch shape. - sdk/metric/pipeline.go — dispatch into Builder.RawBuffer + isAggregatorCompatible. Tests: four unit tests covering the core contract (every sample emitted, second collect is empty i.e. no cumulative semantics retained, per-series cap honoured, many-attribute-set isolation). Design notes: - Both delta and cumulative paths call the same collect(). Raw-buffer has no meaningful cumulative semantics — re-emitting all history every tick would be useless — so the buffer always resets after collect regardless of requested temporality. - We cap per-series, not globally, because the L (label-projection) axis sweep intentionally drives cardinality down; a global cap would couple the two axes. Pre-existing hllsketch build break fixed in passing: - sketchlib-go renamed HyperLogLog.Insert → InsertValue (float64 arg) and Estimate → EstimateCardinality. Two-line patch in hllsketch.go to restore a green build. The sketchlib-go call surface is unchanged at semver zero, so this is mechanical drift, not a behaviour change. Stacked on docs/sdk-three-axis-framework — review that first for the design context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
* docs: RCA — "N=10 throughput collapse" is not a bottleneck PR #185 flagged a universal ~2k pts/s floor at N=10 across all six baselines and attributed it to a coordinated throttle (OTLP SDK / Docker userland-proxy / kernel socket buffers). Rate-invariance test disproves the bottleneck reading. Holding cardinality=1000 and varying EXPORTER_RATE from 1000 to 10000 (10× change) on the same b0a-raw-stream N=1 stack: gateway rate stays flat at 2,000 pts/s and backend rate at 2,001 pts/s. If any of the candidate bottlenecks were the cause, 10× input would produce observable throughput delta — it doesn't. Root cause: PR #182 (`feat(workload): fake-exporter trace-replay mode`) changed the OTel MeterProvider's PeriodicReader interval from `time.Second / time.Duration(rate)` to `time.Second` fixed. With a 1 s interval the SDK pre-aggregates Counter.Add and Gauge.Record calls per attribute set within each tick, so the export rate becomes `cardinality × #instruments × (1/interval)` = 1000 × 2 × 1 = 2000 pts/s, independent of input rate. N=10 is just 10 concurrent producers each correctly emitting 2k. Bigger implication: the paper §6.2 "raw vs sketch bandwidth" story is more fragile than it looks. With the current fake-exporter, the "raw" baselines (B0a / B0b / B1) are already SDK pre-aggregated at the producer — they are not emitting per-sample traffic. The bandwidth delta vs sketch baselines measures payload shape, not "raw samples vs summary per window". Lays out four paper-story options (A revert interval / B dual- interval / C bypass SDK aggregation / D reframe §6.2 as bytes- per-window). Recommends B + D. Does not land a fix; that's a follow-up PR that needs a paper-framing decision. Artifacts: - docs/n10-bottleneck-rca.md — full write-up - deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv — the two-row evidence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: three-axis SDK aggregation framework + sync stale docs Formalizes the SDK-side decision point as a (W, L, agg_type) triple: time window × label projection × encoding. All three axes are independent and correspond one-to-one to what the controller's planner emits per metric. - docs/sdk-aggregation-three-axis-design.md (new) — authoritative design doc for the framework. Maps to existing Mode 1/2/3 vocabulary in delta-transmission-design.md. Enumerates the outstanding SDK-side aggregators (raw-buffer + 5 delta variants) and the SDK runtime hook (hot-reload AttributeFilter) needed to exercise the full planner loop. - docs/paper-outline.md §6 — split §6.2 into four sub-sweeps (a/b/c/d) along the three axes, introduce §6.5 planner-quality as an independent experiment, update claims table to express bandwidth reduction as a three-factor product. - PROGRESS.md — bump date to 2026-04-23; point at new design doc; enumerate outstanding aggregators (~150 LOC raw-buffer + ~500 LOC five delta variants) + fake-exporter knobs + measure-baseline producer-side columns. - docs/n10-bottleneck-rca.md — prepend postscript noting the Options A–D recommendation section is superseded by the three-axis design. Diagnosis content unchanged. - TODO.md (top-level) — retire "N=10 throughput collapse" as P0 blocker (the RCA closed it); replace with the concrete implementation punch list for the three-axis framework. - deploy/TODO.md — add producer-side measurement requirement (producer_cpu_cores / _rss_mib / _bytes_out_per_s) to the instrumentation P1 list; the three-axis sweeps read these from fake-exporter container directly rather than inferring from gateway counters. Includes the n10-bottleneck-rca.md content from the earlier diag/n10-bottleneck-rca branch (superseded by this PR — close that one). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: correct scope — delta-encoding is a flag on existing aggregators Reading opentelemetry-go-patch/sdk/metric/aggregation.go shows that DeltaTransmission is already a field on DDSketch / CountSketch / CountMinSketch / HLLSketch aggregators (landed 2026-03-14), so there's no need to ship five new AggregationDelta<X> types. Only KLL lacks delta support, and its multi-level sample-buffer structure doesn't admit a naive byte-diff — separate design problem, not a §6.2 blocker. Updates docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, and TODO.md to reflect this. Real remaining gap: AggregationRawBuffer (~150 LOC). Proceeding to implement that next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(sdk/metric): AggregationRawBuffer — raw-sample encoding slot Adds AggregationRawBuffer to the SDK as the encoding-axis baseline for the three-axis framework (see docs/sdk-aggregation-three-axis-design.md): where every Counter.Add / Gauge.Record call becomes its own NumberDataPoint on the wire instead of being reduced to a Sum or LastValue per attribute set per tick. Implementation: - sdk/metric/aggregation.go — new AggregationRawBuffer{MaxEventsPerSeries} public type, wired into the Aggregation interface and err() validation alongside the existing sketch aggregators. - sdk/metric/internal/aggregate/rawbuffer.go — aggregator impl. Per attribute.Distinct key, appends (ts, value) to a bounded slice in measure(); in collect() emits each buffered sample as its own metricdata.DataPoint[N] inside a Gauge[N], then clears the buffer. Overflow on a single series is silently dropped with a per-series drop counter (exposing that via a side-channel metric is a follow-up tracked in PROGRESS.md). - sdk/metric/internal/aggregate/aggregate.go — Builder.RawBuffer method, mirroring the KLLSketch / HLLSketch shape. - sdk/metric/pipeline.go — dispatch into Builder.RawBuffer + isAggregatorCompatible. Tests: four unit tests covering the core contract (every sample emitted, second collect is empty i.e. no cumulative semantics retained, per-series cap honoured, many-attribute-set isolation). Design notes: - Both delta and cumulative paths call the same collect(). Raw-buffer has no meaningful cumulative semantics — re-emitting all history every tick would be useless — so the buffer always resets after collect regardless of requested temporality. - We cap per-series, not globally, because the L (label-projection) axis sweep intentionally drives cardinality down; a global cap would couple the two axes. Pre-existing hllsketch build break fixed in passing: - sketchlib-go renamed HyperLogLog.Insert → InsertValue (float64 arg) and Estimate → EstimateCardinality. Two-line patch in hllsketch.go to restore a green build. The sketchlib-go call surface is unchanged at semver zero, so this is mechanical drift, not a behaviour change. Stacked on docs/sdk-three-axis-framework — review that first for the design context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(fake-exporter): three-axis SDK knobs + widened label schema Rewrites the fake-exporter to expose the three-axis knobs defined in docs/sdk-aggregation-three-axis-design.md. The paper's §6.2 sub-sweeps (time / label / encoding) drive all three via env: EXPORTER_SDK_WINDOW PeriodicReader interval (default 15s). Paper's W axis. EXPORTER_SDK_PROJECTION Comma-separated attribute keys to keep; "" = keep all, "-" = drop all. Paper's L axis, implemented via sdkmetric.View AttributeFilter. EXPORTER_SDK_AGG Aggregator kind. Paper's encoding axis. Supported: default | sum | raw-buffer | dd-full | dd-delta | kll | cms-full | cms-delta | cs-full | cs-delta | hll-full | hll-delta Paper raw-baseline story now wires the SDK-native AggregationRawBuffer (landed in feat/aggregation-raw-buffer), so every Counter.Add and Gauge.Record becomes its own NumberDataPoint on the wire at each window flush — no more conflating the encoding axis with the per-tick SDK aggregation behaviour. Workload changes: - Drops `EXPORTER_RATE` (was a no-op under SDK aggregation; see docs/n10-bottleneck-rca.md). Warns loudly when set so stale compose files surface. - Adds `EXPORTER_FREQ_HZ` (default 10 Hz per series) as the app-layer event rate. Orthogonal to SDK_WINDOW. - Widens the synthetic label schema from 2 dims (zone, pod) to 4 dims (zone, rack, node, pod). Default max cardinality is now 4 × 10 × 25 × 10 = 10000; every EXPORTER_*_VALS override is a separate env. The L-axis sweep needs at least 3 dims to cover {full, 3-of-4, 2-of-4, 1-of-4, 0-of-4}. - One goroutine per series, each ticking at period = 1/FREQ_HZ. Cleaner isolation than the old shared ticker + mu.Lock pattern. Build: - go.mod now pins v1.41.0 and replaces every otel sibling at ../../opentelemetry-go (the combined upstream + patch tree produced by restore_opentelemetry_go_patches.sh). Replaces must live here because the patched sdk/metric module's own replaces don't apply when this exporter is the main module. - Dockerfile.fake-exporter now takes sketchlib-go via BuildKit --build-context (no submodule hop), copies in the opentelemetry-go combined tree, and rewrites the sketchlib-go path in go.mod for /src. - Locally verified: go build clean, image builds clean, binary starts and parses all three knobs + deprecation warning. Follow-up: measure-baseline.py producer-side columns, then §6.2 sweeps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 23, 2026
- Delete docs/n10-bottleneck-rca.md — the finding (PR #185's 'N=10 throughput collapse' is SDK pre-aggregation, not a bottleneck) has propagated into the docs that still need it (TODO §1 and the SDK cost evaluation doc summarise the conclusion inline). The long write-up was one-shot debug context, not a reference. - Drop the dangling link from docs/sdk-cost-evaluation.md's header block (and the 'Authoritative for §6.2…' framing — redundant with the doc content). - Fix the other references that pointed at the deleted file: TODO.md §1, PROGRESS.md outstanding-aggregators note, fake-exporter main.go comment and EXPORTER_RATE deprecation log, docker-compose/base.yml comment. Each now explains the point inline. - Rework the kll-delta status in both docs/sdk-cost-evaluation.md and PROGRESS.md from '❌ not yet' → '**no need**'. kll-full covers the encoding-axis comparison; implementing a KLL-specific delta would need a different strategy than the byte-diff the other four sketches use, and the payoff doesn't justify it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Apr 30, 2026
* eval(§6.2): first three-axis sweep pass + findings write-up
Runs the §6.2 a/b/c sub-sweeps defined in
docs/sdk-aggregation-three-axis-design.md at CARDINALITY=1000,
FREQ_HZ=10, N=1, SOAK_S=180, BYTES_WIN=20. 17 cells total,
~65 min wall clock.
Artifacts:
- deploy/scripts/run-paper-6.2-sweeps.sh — wrapper that runs
all three sub-sweeps at named env points and drops CSVs.
- deploy/eval-results/three-axis/6.2a-time-20260423.csv
- deploy/eval-results/three-axis/6.2b-label-20260423.csv
- deploy/eval-results/three-axis/6.2c-encoding-20260423.csv
plus .log files for each sub-sweep stderr.
- deploy/eval-results/three-axis/FINDINGS-20260423.md — results
interpretation with three sections:
- Clean findings (encoding-axis ratios, label-axis RSS drop)
- Methodology caveat (BYTES_WIN < W under-counts bytes)
- Surprises worth re-checking (§6.2b CPU-up-with-projection,
cms-delta memory overhead)
Key clean findings:
- §6.2c encoding: raw-buffer holds 108 MiB RSS vs 33-39 MiB for
sketch encodings (~3× memory cost for per-event storage) and
emits ~7× the wire bytes of any sketch. cms-delta costs 2×
the RSS of cms-full because it holds prev-tick snapshot for
diff.
- §6.2b label: projecting to zone,rack cuts producer RSS from
66.8 → 33.7 MiB as attribute-set folding reduces sketch
instance count.
Known caveat: BYTES_WIN=20s < W=60s in most cells, so
producer_bytes_out_per_s is under-reported by factor ~W/BYTES_WIN.
Ratios within a sub-sweep (all at same W) are unaffected. V2
rerun with BYTES_WIN ≥ 2W will fix absolute numbers — doc'd in
FINDINGS as the top-priority follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: rewrite FINDINGS to describe the experiment self-contained
Drops paper-section references from the findings doc so it reads
as experimental results standing on their own, not as "§6.2
sub-sweep output". Structure is now:
1. Experiment design — what the (W, L, agg_type) triple means,
what each sub-experiment varies vs holds fixed, how each
cell is measured.
2. Methodology caveat (unchanged — BYTES_WIN < W under-reports).
3. Results per sub-experiment (encoding axis, label axis,
time axis) with the numbers and what to make of them.
4. Non-obvious findings worth a second look.
5. Next actions.
No numeric content changed; CSVs untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: rename findings title to 'SDK aggregation experiments'
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: explain W + push vs pull in FINDINGS
Adds a 'What W actually is (and push vs pull)' subsection right
after the (W, L, agg_type) definition so the doc doesn't assume
the reader knows OTel's Reader/PeriodicReader model. Compares
OTLP push and Prometheus pull aggregation-equivalence and
calls out the two places they actually differ for this project
(NAT-friendliness + controller-reconfig channel).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: mark AggregationRawBuffer as landed (#189)
docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, TODO.md
all still had AggregationRawBuffer flagged ❌. It shipped in
#189 back in April — updating the status rows + cross-referencing
the contract test (deploy/fake-exporter/sdk_emit_test.go) +
the implementation path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: expand AggregationRawBuffer design — why + what
Rewrites the AggregationRawBuffer subsection in the design doc to
explain *why* raw-buffer is a first-class encoding slot (not just
what it does). Key points the previous version didn't make:
- Every other upstream Aggregation is lossy by design; raw-buffer
is the only SDK-native option that preserves the full
observation stream. Without it there's no reference point
for sketch-vs-raw claims.
- It's also the encoding that can serve queries no summary
can: exact events for cold-fallback replay, per-sample audit,
downstream sketch computation the SDK policy didn't
anticipate.
- Memory cost scales as O(W × event_rate × cardinality) —
the tradeoff the encoding axis measures.
Contract + overflow + temporality specifics kept from the
earlier version and folded into dedicated subsections.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: rename to 'SDK aggregation cost' + strip landed-date stamps
- Rename `docs/sdk-aggregation-three-axis-design.md` →
`docs/sdk-aggregation-cost.md`. Title updated to 'SDK
aggregation cost'. Old name was a framework description;
new name says directly what the doc is measuring.
- Update every cross-reference in docs, PROGRESS.md, TODO.md,
deploy/ (compose, Dockerfile, go.mod, main.go, scripts,
FINDINGS) to point at the new filename.
- Drop `✅ landed 2026-XX-XX` and `(#189)` stamps in status
tables and TODO checkboxes. Merged to main = ✅; the dates
and PR numbers are git-log territory, not running-doc
territory.
No substantive content change; only the framing/file name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove stray __pycache__ + ignore it
* docs: rename to 'SDK cost evaluation' to signal this is the eval doc
File: docs/sdk-aggregation-cost.md → docs/sdk-cost-evaluation.md
Title: 'SDK cost evaluation'
All cross-references in docs/, PROGRESS.md, TODO.md, deploy/
updated. No content change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: delete n10-bottleneck-rca.md; rework kll-delta status to 'no need'
- Delete docs/n10-bottleneck-rca.md — the finding (PR #185's
'N=10 throughput collapse' is SDK pre-aggregation, not a
bottleneck) has propagated into the docs that still need it
(TODO §1 and the SDK cost evaluation doc summarise the
conclusion inline). The long write-up was one-shot debug
context, not a reference.
- Drop the dangling link from docs/sdk-cost-evaluation.md's
header block (and the 'Authoritative for §6.2…' framing —
redundant with the doc content).
- Fix the other references that pointed at the deleted file:
TODO.md §1, PROGRESS.md outstanding-aggregators note,
fake-exporter main.go comment and EXPORTER_RATE deprecation
log, docker-compose/base.yml comment. Each now explains the
point inline.
- Rework the kll-delta status in both docs/sdk-cost-evaluation.md
and PROGRESS.md from '❌ not yet' → '**no need**'. kll-full
covers the encoding-axis comparison; implementing a KLL-specific
delta would need a different strategy than the byte-diff the
other four sketches use, and the payoff doesn't justify it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: strip paper-section references; rename sweep driver; drop follow-up list
Across-the-board cleanup so evaluation docs stand on their own and
don't cite paper section numbers. Evaluations describe design +
methodology + results directly; paper framing belongs only in
docs/paper-outline.md.
- Rename deploy/scripts/run-paper-6.2-sweeps.sh →
run-sdk-cost-sweeps.sh. Filename drops the §6.2 embedding.
- Delete docs/design-stateful-protocol.md's "(Paper §4.5)" title
suffix; title is now just the protocol name.
- docs/sdk-cost-evaluation.md:
- remove the obsolete "Implementation order (follow-up PRs)"
section — most items are merged, and the remaining one
lives in TODO.md anyway
- remove the "Non-goals of this doc" section — pointer list
was stale
- drop the paper-§6 framing in the opening; doc now reads
as standalone evaluation design
- rewrite the ablation table: instead of "6.2a Time axis /
6.2b Label axis / …" with "Claim" column, plain "Time /
Label / Encoding / Combined" with "What the sweep
measures" column
- strip the few remaining §6.2 / §6.5 inline refs
- Strip paper-§N from every config / compose / script / TODO /
PROGRESS comment where it appeared. Specifically:
- PROGRESS.md outstanding-aggregators section
- TODO.md §1 header (now "SDK cost evaluation") + all inline
§6.2 references
- deploy/TODO.md instrumentation section headings + inline
- deploy/README.md "Paper §6 mapping" → "Evaluation → metric
mapping"; drop "paper's §6 eval" framing
- deploy/configs/*.yaml — comment prefixes like "Paper §6.2
baseline:" → plain "Baseline:"
- deploy/configs/prometheus.yml, grafana-datasources.yml —
same
- deploy/docker-compose/*.yml — "Paper §6" → "Scale dials"
etc.
- deploy/helm/asap/{values,Chart}.yaml — same
- deploy/fake-exporter/main.go — "Paper's W axis" / "L axis"
/ "encoding axis" → "Time axis" / "Label axis" / "Encoding
axis"
- deploy/fake-exporter/sdk_emit_test.go — §6.2c → encoding-axis
- Left in place: internal design-doc cross-references like
"delta-transmission-design.md §16.2" where the §N refers to a
section of a design doc rather than the paper. Those are legit
technical writing, not paper citations.
No substantive content or behaviour change anywhere; pure
framing/naming cleanup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* eval: rename artifacts to match SDK cost evaluation naming
- deploy/eval-results/three-axis/ → deploy/eval-results/sdk-cost/
to match the design doc (docs/sdk-cost-evaluation.md).
- CSV + log files lose the paper-section prefix:
6.2a-time-20260423 → time-axis-20260423
6.2b-label-20260423 → label-axis-20260423
6.2c-encoding-20260423 → encoding-axis-20260423
- Update every reference in
docs/sdk-cost-evaluation.md, TODO.md,
deploy/scripts/run-sdk-cost-sweeps.sh, and the FINDINGS doc.
Data unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: sharpen raw-buffer rationale in SDK cost eval
The previous 'Why this baseline exists' blurb framed raw-buffer as
a generic reference point but didn't pin down (a) that OTel's
native aggregators are lossy *specifically over the emit period
W*, or (b) that the experiment's value comes from running
raw-buffer at the *same* W as the sketch encodings so the
comparison is apples-to-apples.
Rewrite leads with those two observations, then closes on the
'without it, the ablation collapses into sketch vs nothing' line.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: explain Aggregation<X>Delta as a design decision, not a TODO
The previous 'Adding Aggregation<X>Delta (×5)' block read like a
PR checklist — semantics / encoding / LOC estimate. This is a
design doc, not a task tracker.
Rewrite leads with *what* delta encoding is (full state vs. sparse
diff of changed cells; aggregator keeps prev-tick state; full
re-emit on L / param change) and *why* it deserves a distinct
slot in the encoding-axis evaluation:
1. The bandwidth claim 'sketch < raw' has two independent
factors (sketch payload vs raw, and delta vs full sketch).
Measuring *-full and *-delta as separate rows on the
encoding axis lets each factor be attributed directly.
2. Delta is a memory-for-bandwidth trade: the aggregator
holds prev-tick state alongside current, roughly doubling
RSS. That tradeoff belongs next to the bandwidth savings
in the encoding row, not buried as an implementation note.
Also notes that KLL is intentionally absent from the delta column
and points at the existing note above for the rationale.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: shorten section headings
AggregationRawBuffer design → AggregationRawBuffer
Aggregation<X>Delta — what it is and why it's a separate slot →
Aggregation<X>Delta
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* deploy: rename run-three-axis-sweep.sh → run-sdk-cost-sweep.sh
Last paper-shaped artefact in the cost-evaluation tooling. The
inner grid driver is now 'run-sdk-cost-sweep.sh' (singular —
runs one sweep over the grid the caller specifies), and the
outer wrapper stays 'run-sdk-cost-sweeps.sh' (plural — runs the
three canonical sub-sweeps).
Updates every reference in the evaluation doc, findings, TODO,
and the wrapper script itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* deploy: rename sweep scripts by role (grid vs eval)
- run-sdk-cost-sweep.sh → run-sdk-cost-grid.sh
Runs a W × L × agg grid the caller specifies via env.
- run-sdk-cost-sweeps.sh → run-sdk-cost-eval.sh
Runs the three canonical sub-sweeps (time / label / encoding)
that make up the SDK cost evaluation; calls run-sdk-cost-grid
three times with different fixed/swept axes.
The singular-vs-plural distinction was too subtle; 'grid' vs
'eval' spells out the role difference directly.
Updates references in docs/sdk-cost-evaluation.md, FINDINGS,
TODO.md, and the eval script's own call site.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: merge TODO.md into PROGRESS.md
Two top-level docs were covering the same ground — PROGRESS
drifted into historical session notes, TODO duplicated the
'what's landed' summary and added 'what's left'. Consolidated
into a single PROGRESS.md with three sections:
- Implemented — ground-truthed against current code
- Outstanding — paper blockers + non-blocker SDK work
- Future work — post-paper
Top-level TODO.md deleted. Cross-references in deploy/TODO.md
and deploy/README.md updated to point at PROGRESS.md instead
(the local 'See TODO.md in this directory' ref in README refers
to deploy/TODO.md, kept as-is).
Sketch-processor and aggregator status tables verified against
the patch trees:
opentelemetry-go-patch/sdk/metric/aggregation.go lists all six
Aggregation types we document (DDSketch/KLL/CS/CMS/HLL +
RawBuffer), matching the table.
opentelemetry-collector-contrib-patch/processor/ has the five
per-sketch processors + two new *mergeprocessor variants
(countsketchmerge, countminsketchmerge) surfaced in the
table.
deploy/helm/asap/ has Chart.yaml + values.yaml, no templates/
dir — reflected honestly.
No semantic change; pure consolidation + refresh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: merge deploy/TODO.md into PROGRESS.md
deploy/TODO.md was ~90% duplicated with PROGRESS.md's
'Outstanding' section — same instrumentation, query-side,
fault-injection, and Helm items, rephrased. Consolidate.
Genuinely unique content folded into PROGRESS.md:
- Helm template landing order (file-by-file sequence, from
_helpers.tpl through prometheus.yaml/grafana.yaml) — now
part of the 'Helm chart templates' future-work bullet.
- Fault-injection script names (controller-kill.sh,
agent-kill.sh, network-partition.sh) — now bullet-listed
under paper blocker #8.
- Compose polish (per-agent AGENT_ID label, CI check,
deploy/k8s/ plain-manifest alternative) — new bullet under
Future work; didn't previously live anywhere else.
Fixed three dangling refs to deploy/TODO.md:
- deploy/README.md — the Helm 'see TODO.md in this directory'
line now points at ../PROGRESS.md.
- deploy/docker-compose/base.yml header comment — 'tracked
in deploy/TODO.md' → 'tracked in PROGRESS.md'.
- PROGRESS.md's own sibling-docs list drops the now-dead
deploy/TODO.md entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: base.yml comment ref to deploy/TODO.md → PROGRESS.md
Follow-up to previous commit which tried to update this line but
lost it to a read-modify-write race. Pure comment fix.
* eval(sdk-cost): profile per-sample vs batched raw-buffer emit
Answers the follow-up: at the same ground-truth raw-sample
rate (10 Hz per series), does it cost more or less to emit
each sample per tick (W=100ms) vs buffer them and ship in a
batch (W=10s)?
Knob changes:
- deploy/fake-exporter/main.go — new EXPORTER_PPROF_ADDR env
var. When set, opens net/http/pprof on the given address so
external tools can sample CPU / heap / goroutines live.
No-op when unset; production runs stay unchanged.
Findings (full detail in
deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md,
4 configs × 15s pprof captures against the live gateway):
- At matching cardinality, per-tick emit (W=100ms) uses ~2×
the CPU of batched (W=10s). The delta isn't data volume —
it's the per-flush overhead (gRPC ClientConn.Invoke + flate
writer init) that W=100ms pays 100× more often.
- Per-tick emit keeps RSS flat. Batched W=10s with card=100
spikes +39 MiB at flush — the protobuf transform
intermediates (transform.Value / KeyValue / DataPoints)
dominate heap.
- Config A (per-tick) CPU hotspots: PeriodicReader.collectAndExport
→ otlpmetricgrpc.Export → grpc.ClientConn.Invoke (54 % cum).
- Config B (batched) CPU hotspots at card=100:
runtime.gcBgMarkWorker (30 %) + runtime.gcDrain (30 %) —
GC chases the flush burst.
- Neither scales linearly where you'd expect: A's CPU scales
with 1/W (flush frequency), B's memory scales with
cardinality × samples_per_window.
- Design corner: small W + modest cardinality is cheap; large
W + large cardinality is dangerous (memory bursts that
container cgroups will clip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* eval(sdk-cost): fine-grained pprof — mutex contention dominates at 1M events/s
Extends the emit-cadence profile to fine-grained scale:
cardinality=1000 × freq_hz ∈ {100, 1000}, so 100 k and 1 M
events/s. Runs the same A(W=100ms) vs B(W=10s) comparison
as the first pass.
Three findings not visible in the small-scale pass:
1. A-vs-B CPU gap collapses. At tiny scale A was ~2× B
(per-flush fixed cost dominated). At fine scale per-flush
overhead amortises over 10k–100k dp batches and the gap
disappears (A:2.8c vs B:3.0c at 100k/s; A:8.1c vs B:9.8c
at 1M/s — B slightly worse).
2. Hot path shifts from gRPC export to aggregator mutex
contention. rawBufferValues.measure takes a single
valuesMu across the whole attribute map. With 1000
goroutines firing 1000 Adds/s = 1M Lock acquisitions/s,
lockSlow dominates (~26% of CPU). This is a real design
issue — a sharded buffer or per-series lock is needed
before raw-buffer is usable at ≥1M events/s.
3. Heap is dominated by OTLP transform intermediates, not
the raw-buffer. At 1M events/s ~1 GiB of KeyValue/Value/
DataPoints objects are allocated per flush in both
configs. Pool reuse in the upstream transform would help.
Results table + hot-path diff + heap breakdown appended to
deploy/eval-results/sdk-cost/PROFILE-sdk-emit-cadence-20260423.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full 6-baseline sweep at N=10. Per-agent throughput collapses to ~2,000 pts/s across every baseline, vs 130k–326k at N=1.
Gateway end-to-end = 10 × 2k = 20k/s for raw baselines. Agents idle (0.01c CPU, ~220 MiB RSS) → system bottleneck, not per-baseline. B0 raw forward affected identically to B3 delta sketch.
Likely cause: fake-exporter OTLP/gRPC SDK backpressure — 10 producers × nominal 1M pts/s overwhelms some shared limit. Per-producer degradation (150×) + aggregate degradation (16× vs N=1) is consistent with a per-process quota in the producer path.
Paper implication: N=10 row is measuring producer bottleneck, not sketch pipeline. Options:
Archiving the data honestly. Diagnosis follow-up.
🤖 Generated with Claude Code