docs: three-axis SDK aggregation framework + sync stale docs - #188
Merged
Merged
Conversation
PR #185 flagged a universal ~2k pts/s floor at N=10 across all six baselines and attributed it to a coordinated throttle (OTLP SDK / Docker userland-proxy / kernel socket buffers). Rate-invariance test disproves the bottleneck reading. Holding cardinality=1000 and varying EXPORTER_RATE from 1000 to 10000 (10× change) on the same b0a-raw-stream N=1 stack: gateway rate stays flat at 2,000 pts/s and backend rate at 2,001 pts/s. If any of the candidate bottlenecks were the cause, 10× input would produce observable throughput delta — it doesn't. Root cause: PR #182 (`feat(workload): fake-exporter trace-replay mode`) changed the OTel MeterProvider's PeriodicReader interval from `time.Second / time.Duration(rate)` to `time.Second` fixed. With a 1 s interval the SDK pre-aggregates Counter.Add and Gauge.Record calls per attribute set within each tick, so the export rate becomes `cardinality × #instruments × (1/interval)` = 1000 × 2 × 1 = 2000 pts/s, independent of input rate. N=10 is just 10 concurrent producers each correctly emitting 2k. Bigger implication: the paper §6.2 "raw vs sketch bandwidth" story is more fragile than it looks. With the current fake-exporter, the "raw" baselines (B0a / B0b / B1) are already SDK pre-aggregated at the producer — they are not emitting per-sample traffic. The bandwidth delta vs sketch baselines measures payload shape, not "raw samples vs summary per window". Lays out four paper-story options (A revert interval / B dual- interval / C bypass SDK aggregation / D reframe §6.2 as bytes- per-window). Recommends B + D. Does not land a fix; that's a follow-up PR that needs a paper-framing decision. Artifacts: - docs/n10-bottleneck-rca.md — full write-up - deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv — the two-row evidence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Formalizes the SDK-side decision point as a (W, L, agg_type) triple: time window × label projection × encoding. All three axes are independent and correspond one-to-one to what the controller's planner emits per metric. - docs/sdk-aggregation-three-axis-design.md (new) — authoritative design doc for the framework. Maps to existing Mode 1/2/3 vocabulary in delta-transmission-design.md. Enumerates the outstanding SDK-side aggregators (raw-buffer + 5 delta variants) and the SDK runtime hook (hot-reload AttributeFilter) needed to exercise the full planner loop. - docs/paper-outline.md §6 — split §6.2 into four sub-sweeps (a/b/c/d) along the three axes, introduce §6.5 planner-quality as an independent experiment, update claims table to express bandwidth reduction as a three-factor product. - PROGRESS.md — bump date to 2026-04-23; point at new design doc; enumerate outstanding aggregators (~150 LOC raw-buffer + ~500 LOC five delta variants) + fake-exporter knobs + measure-baseline producer-side columns. - docs/n10-bottleneck-rca.md — prepend postscript noting the Options A–D recommendation section is superseded by the three-axis design. Diagnosis content unchanged. - TODO.md (top-level) — retire "N=10 throughput collapse" as P0 blocker (the RCA closed it); replace with the concrete implementation punch list for the three-axis framework. - deploy/TODO.md — add producer-side measurement requirement (producer_cpu_cores / _rss_mib / _bytes_out_per_s) to the instrumentation P1 list; the three-axis sweeps read these from fake-exporter container directly rather than inferring from gateway counters. Includes the n10-bottleneck-rca.md content from the earlier diag/n10-bottleneck-rca branch (superseded by this PR — close that one). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reading opentelemetry-go-patch/sdk/metric/aggregation.go shows that DeltaTransmission is already a field on DDSketch / CountSketch / CountMinSketch / HLLSketch aggregators (landed 2026-03-14), so there's no need to ship five new AggregationDelta<X> types. Only KLL lacks delta support, and its multi-level sample-buffer structure doesn't admit a naive byte-diff — separate design problem, not a §6.2 blocker. Updates docs/sdk-aggregation-three-axis-design.md, PROGRESS.md, and TODO.md to reflect this. Real remaining gap: AggregationRawBuffer (~150 LOC). Proceeding to implement that next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Formalizes the SDK-side decision point as a
(W, L, agg_type)triple — time window × label projection × encoding. All three axes are independent and correspond one-to-one to what the controller's planner emits per metric.docs/sdk-aggregation-three-axis-design.md— authoritative design doc. Maps to existing Mode 1/2/3 vocabulary indelta-transmission-design.md. Enumerates outstanding SDK-side aggregators.docs/paper-outline.md §6— split §6.2 into four sub-sweeps (a/b/c/d) along the three axes; new §6.5 planner-quality experiment; claims table updated to express bw reduction as a three-factor product.PROGRESS.md— bump date, point at the new design doc, enumerate outstanding aggregators (AggregationRawBuffer+ thekll-deltagap; delta for DD/CS/CMS/HLL already exists as a flag on the*-fulltypes).docs/n10-bottleneck-rca.md— postscript marks Options A–D as superseded; diagnosis content kept.TODO.md,deploy/TODO.md— retire stale items; add three-axis implementation punch list + producer-side measurement requirement.Subsumes earlier
diag/n10-bottleneck-rcaPR — please close that one. All RCA content is included here.Test plan
docs/sdk-aggregation-three-axis-design.md— confirm(W, L, agg_type)framing matches intentpaper-outline.mdmatches what you want to measuredelta-transmission-design.mdMode 1/2/3) is accurate🤖 Generated with Claude Code