diff --git a/PROGRESS.md b/PROGRESS.md index 5dd3c300..7a8f09a7 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,6 +1,14 @@ # DataCollector — Implementation Progress -_Last updated: 2026-03-14 (SDK pre-aggregation update)_ +_Last updated: 2026-04-23 — three-axis SDK framework formalized_ + +See [`docs/sdk-aggregation-three-axis-design.md`](docs/sdk-aggregation-three-axis-design.md) +for the current authoritative design of the SDK decision point +(time window `W` × label projection `L` × encoding `agg_type`). +The 2026-03-14 SDK pre-aggregation batch below covers the +`*-full` encoding column; the `raw-buffer` and `*-delta` +columns are still open (see "Outstanding SDK aggregators" +section at the bottom of this file). --- @@ -227,3 +235,54 @@ OpenTelemetry Collector (custom build via OCB) Prometheus / Grafana Metric: _hll_cardinality{host="...", metric="..."} ``` + +--- + +## 2026-04-23 update — three-axis SDK framework + +The five SDK pre-aggregation aggregators above (DDSketch / KLL / +CountSketch / CountMinSketch / HLLSketch) all implement the +`*-full` encoding slot of the three-axis `(W, L, agg_type)` +framework defined in +[`docs/sdk-aggregation-three-axis-design.md`](docs/sdk-aggregation-three-axis-design.md). + +### Outstanding SDK aggregators (P1 for paper §6.2) + +Correction after a read of +`opentelemetry-go-patch/sdk/metric/aggregation.go` — **delta +encoding is already a flag on the four sparse-state sketches** +(`DeltaTransmission: true`), not a separate aggregator. So the +real gap is smaller than the earlier plan: + +| Aggregator | Slot | Status | Notes | +|---|---|---|---| +| `AggregationRawBuffer` | `agg_type=raw-buffer` | ❌ | Buffers `(ts, attrs, value)` tuples within `W`, emits batch of `NumberDataPoint`s per tick. Overflow: drop + drop-counter metric. ~150 LOC. | +| `AggregationKLLSketch.DeltaTransmission` | `agg_type=kll-delta` | ❌ | KLL's multi-level sample buffers don't support a natural byte-diff; adding delta requires exposing per-level internals from `sketchlib-go` or shipping incremental adds. **Not a §6.2 blocker** (see design doc for rationale). | + +The other four sketch delta slots (DDSketch / CountSketch / +CountMinSketch / HLLSketch) already work via the +`DeltaTransmission: true` flag from the 2026-03-14 batch above. + +### Outstanding SDK runtime support + +- **Hot-reload of View `AttributeFilter`** — required for the + controller-in-loop §6.5 scenario where the planner pushes a + new projection `L` mid-run. Upstream OTel Go SDK doesn't + support replacing a View's filter after MeterProvider + construction; needs a small patch in + `opentelemetry-go-patch/sdk/metric/` to expose a swap API. + Not a §6.2 blocker (each static sweep run is a fresh + process). + +### Downstream dependents + +- `deploy/fake-exporter/main.go` — needs to drop + `EXPORTER_RATE` (semantically meaningless now — see + [`docs/n10-bottleneck-rca.md`](docs/n10-bottleneck-rca.md)) + and expose `EXPORTER_SDK_WINDOW`, `EXPORTER_SDK_PROJECTION`, + `EXPORTER_SDK_AGG`. Widen the synthetic label schema from 2 + dims (`{zone, pod}`) to 4 dims (`{zone, rack, node, pod}`) + so the `L`-axis sweep has range. +- `deploy/scripts/measure-baseline.py` — add producer-side + columns (`producer_cpu_cores`, `producer_rss_mib`, + `producer_bytes_out_per_s`). diff --git a/TODO.md b/TODO.md index 1ff746bb..242bdf8b 100644 --- a/TODO.md +++ b/TODO.md @@ -1,6 +1,6 @@ # TODO — DataCollector + controller for paper submission -_Last updated: 2026-04-23 (post N=10 sweep, PRs #168–#185)_ +_Last updated: 2026-04-23 — N=10 "collapse" diagnosed + new 3-axis SDK design landed_ This doc lists what's left to get a VLDB / SIGMOD submission out the door. For the v1 paper we keep updating `controller/` @@ -43,35 +43,54 @@ landed over PRs #168–#185. Briefly: ## For paper submission (blocker) -### 1. Multi-agent scale — N=10 throughput collapse (P0) - -N=10 sweep (#185, 2026-04-22) surfaced a system-wide bottleneck: -per-agent throughput collapses from 130k–326k pts/s at N=1 to a -universal ~2k floor at N=10, identical across all six baselines -(so it's a topology / producer-SDK issue, not a sketch pipeline -issue). Agents are near-idle (0.01c, 220 MiB). Gateway sees -10 × 2k = 20k total, i.e. coordinated throttling. - -Until this is diagnosed and fixed, the paper's "scales linearly -N ∈ {1, 10, 100}" claim is unsupported. - -Candidate causes, ranked: - -1. **fake-exporter OTLP/gRPC SDK backpressure.** 10 producers × - 1M nominal points/s (rate × cardinality) likely overwhelms a - shared socket or SDK reader limit. First thing to try: tune - `WithMaxQueueSize` / `WithMaxExportBatchSize`, or drop to a - raw gRPC client for the scale sweep. -2. **Docker userland-proxy CPU contention** across many - simultaneous gRPC connections on the default bridge network. - Cheap test: switch to `network_mode: host` for agents + - gateway and re-run. -3. **Kernel socket buffers.** `net.core.somaxconn / - net.core.rmem_max` at stock Ubuntu defaults may cap inbound - gRPC streams. - -Post-fix deliverable: re-run N=1, N=10, N=100 sweeps; paper's -§6.7 "scales linearly" figure uses those three points. +### 1. SDK three-axis aggregation framework (P0) — supersedes the "N=10 throughput collapse" blocker + +The N=10 "collapse" (#185) turned out not to be a bottleneck — +[`docs/n10-bottleneck-rca.md`](docs/n10-bottleneck-rca.md) walks +through the diagnosis. The 2 k pts/s floor was the OTel SDK's +correct pre-aggregation output at `interval=1 s, cardinality=1000, +2 instruments`, independent of input rate. That finding reframed +the paper's §6.2 bandwidth claim as a **three-independent-factor +product** (see +[`docs/sdk-aggregation-three-axis-design.md`](docs/sdk-aggregation-three-axis-design.md)). + +Concrete work items (P0 because §6.2 can't run without them): + +- [ ] **`AggregationRawBuffer`** in + `opentelemetry-go-patch/sdk/metric/aggregation.go` — + `(ts, attrs, value)` buffer, emit batch per tick, drop + + drop-counter on overflow. ~150 LOC + tests. +- [x] ~~`AggregationDelta` × 5~~ — on inspection, four of five + (DDSketch / CS / CMS / HLL) already have `DeltaTransmission` + as a flag on the `*-full` aggregator (2026-03-14 batch). + Only `kll-delta` is missing and is not a §6.2 blocker + (KLL's multi-level buffer structure needs a different + delta strategy — see + [`docs/sdk-aggregation-three-axis-design.md`](docs/sdk-aggregation-three-axis-design.md)). +- [ ] **`fake-exporter` rewrite** (`deploy/fake-exporter/main.go`) + — drop `EXPORTER_RATE`; add `EXPORTER_SDK_WINDOW`, + `EXPORTER_SDK_PROJECTION`, `EXPORTER_SDK_AGG`. Widen label + schema from 2 dims to 4 (`{zone, rack, node, pod}`). +- [ ] **`measure-baseline.py`** producer-side columns — + `producer_cpu_cores`, `producer_rss_mib`, + `producer_bytes_out_per_s` (scrape the fake-exporter + container's cgroup + interface counters). +- [ ] **§6.2 sweeps** at `N=1`: + - 6.2a time: `W ∈ {1s, 15s, 60s, 300s}` × + `L=full, agg=dd-full` + - 6.2b label: `\|L\| ∈ {0,1,2,3,4}` × `W=60s, agg=dd-full` + - 6.2c encoding: `agg ∈ {raw-buffer, dd-full, dd-delta, + kll-full, kll-delta, cms-full, hll-full}` × + `W=60s, L=typical projection` + - 6.2d combined: best per-metric triple vs `raw-buffer + + full-L + W=15s`. +- [ ] **N-scale sweep rerun** at fixed representative + `(W=60s, L=subset, agg=dd-delta)` across `N ∈ {1, 10, 100}`. + This is now the honest scalability test — the 2 k floor + from #185 is expected; we're looking for whether gateway / + backend hold up as aggregate ingress grows. + +Depends on nothing upstream; can start immediately. ### 2. Instrumentation — fill the `nan` columns (P1) diff --git a/deploy/TODO.md b/deploy/TODO.md index 0adfe3b2..b09e0262 100644 --- a/deploy/TODO.md +++ b/deploy/TODO.md @@ -26,9 +26,29 @@ is instrumentation completeness, Helm templates, and polish. ## Instrumentation gaps (P1 — blocks §6.2/6.3 figures) -The N=10 sweep CSV has several `nan` cells that need filling -before the paper plots can be drawn. Tracked top-level as -`DataCollector/TODO.md §2`; implementation lives here. +The §6.2 sub-sweeps defined in +[`../docs/sdk-aggregation-three-axis-design.md`](../docs/sdk-aggregation-three-axis-design.md) +require **producer-side** measurements that don't exist yet; +the N=10 sweep CSV also has several `nan` cells on the +collector side. Both tracked top-level as +`DataCollector/TODO.md §1-ish` (the old §2 wording is rolled +into §1). + +### Producer-side columns (new, P0) + +- [ ] **`producer_cpu_cores`** — `docker stats` or cgroup + read on the `fake-exporter` container. The §6.2 + SDK-side CPU claim is "how much does `agg_type=*-delta` + cost the producer vs `*-full`". +- [ ] **`producer_rss_mib`** — same source. `agg_type=raw-buffer` + is expected to have the largest producer RSS (sample + buffer); we need to measure the knee vs window `W`. +- [ ] **`producer_bytes_out_per_s`** — agent-container + `container_network_transmit_bytes_total{name="fake-exporter"}`, + `rate()` over the measurement window. The main §6.2 + bandwidth axis; we've been inferring this from + gateway-side counters which conflates multiple agents + at N > 1. - [ ] **Byte counters on raw and Gorilla baselines.** `agent_in_kib_per_s` / `agent_out_kib_per_s` are `nan` diff --git a/deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv b/deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv new file mode 100644 index 00000000..d8280c0d --- /dev/null +++ b/deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv @@ -0,0 +1,3 @@ +baseline,scale,rate,cardinality,agent_cpu_cores,agent_rss_mib,agent_in_kib_per_s,agent_out_kib_per_s,agent_points_per_s,gateway_cpu_cores,gateway_rss_mib,gateway_points_per_s,gateway_out_series_per_s,backend_cpu_pct,backend_rss_mib,backend_samples_per_s,backend_query_p99_ms +b0a-raw-stream,N1,1000,1000,nan,nan,nan,nan,nan,0.043,196.652,2000.000,2000.000,nan,nan,2001.000,nan +b0a-rate10k,N1,10000,1000,nan,nan,nan,nan,nan,0.044,196.422,2000.000,2000.000,1.690,11.770,2001.000,nan diff --git a/docs/n10-bottleneck-rca.md b/docs/n10-bottleneck-rca.md new file mode 100644 index 00000000..9cde6241 --- /dev/null +++ b/docs/n10-bottleneck-rca.md @@ -0,0 +1,245 @@ +# RCA: "N=10 throughput collapse" (PR #185) — not a bottleneck + +_Written: 2026-04-23. Supersedes the bottleneck hypothesis in +the PR #185 commit message._ + +> **Status (2026-04-23, later the same day):** the diagnosis +> in this doc stands. The **Options A–D recommendation section +> below is superseded** by +> [`sdk-aggregation-three-axis-design.md`](sdk-aggregation-three-axis-design.md), +> which makes `raw-buffer` / `*-full` / `*-delta` distinct SDK +> aggregators (encoding axis), independent of time window `W` +> and label projection `L`. Options A / B of this doc reduce to +> "revert to a per-tick interval", which isn't a paper strategy — +> it's an accidental way to get near-per-sample emission. +> Option C (bypass SDK) and Option D (reframe as bytes-per-window) +> are both partially realized in the three-axis design: +> bytes-per-window is the metric; `raw-buffer` is the SDK-native +> "every sample" encoding (doesn't bypass the SDK, just picks a +> different aggregator). Read the diagnosis, skip the +> recommendation. + +## TL;DR + +The ~2,000 pts/s per-agent floor observed across all baselines +in `sweep-N10-20260422.csv` is **not** a scale bottleneck. +It is the correct export rate of the fake-exporter's OTel SDK +setup after PR #182 changed the `PeriodicReader` interval from +`1s / rate` (variable) to `1s` (fixed). At `cardinality=1000` +and two instruments (Counter + Gauge), `1000 × 2 × (1/1s) = +2000 data points/s` — independent of `EXPORTER_RATE`. + +Fix is not a kernel / Docker / SDK tuning exercise. Fix is a +paper-story decision: what _should_ the fake-exporter emit, and +what does the §6.2 "bandwidth reduction" claim actually compare? + +## Evidence + +Rate-invariance test — N=1, b0a-raw-stream, held cardinality +constant, varied `EXPORTER_RATE` by 10×: + +| EXPORTER_RATE | EXPORTER_CARDINALITY | `gateway_points_per_s` | `backend_samples_per_s` | +|---:|---:|---:|---:| +| 1,000 | 1,000 | 2,000 | 2,001 | +| 10,000 | 1,000 | 2,000 | 2,001 | + +Both runs: single producer, 60s batch processor, 150s soak, +Prometheus `rate()` over 2m window. Gateway aggregate flat at +2k/s; 10× input-rate delta produces **zero** throughput delta. + +Raw CSV: [`deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv`](../deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv). + +## Root cause + +`deploy/fake-exporter/main.go` — PeriodicReader interval. + +Before PR #182 (git blame: #171/#175 era): + +```go +reader := sdkmetric.NewPeriodicReader(exp, + sdkmetric.WithInterval(time.Second/time.Duration(rate))) +``` + +After PR #182 (current): + +```go +reader := sdkmetric.NewPeriodicReader(exp, + sdkmetric.WithInterval(time.Second)) +``` + +The "before" form set interval = `1s / rate`. At `rate=1000` +the SDK exported every 1 ms, so each tick carried ≤1 ms of +accumulated `counter.Add` / `gauge.Record` calls — effectively +one data point per attribute set per ms → ~2 M pts/s nominal, +bandwidth-bottlenecked down to the ~325 k observed in +`sweep-N1-20260422.csv`. + +The "after" form set interval = `1s` fixed. Within each 1 s +tick, the OTel SDK pre-aggregates: + +- `Counter.Add(…)` calls per attribute set are summed into a + single `Sum` data point per attribute set per tick. +- `Gauge.Record(…)` calls per attribute set are reduced to the + last-recorded value per attribute set per tick. + +So regardless of how many `Add` / `Record` calls the ticker +fires per second, each tick emits exactly `cardinality × +#instruments` data points (1000 × 2 = 2000). `1 tick/s × 2000 += 2000 pts/s`. This is the floor PR #185 saw. + +The N=10 observation of `10 × 2k = 20k` aggregate is not +"coordinated throttling" — it is ten concurrent, independent +producers each emitting the correct pre-aggregated 2k each. + +## Why PR #185's ruled-out causes were ruled out correctly + +PR #185 correctly ruled out: + +- **Agent saturation** (agents at 0.01c, 220 MiB) — right + conclusion. The agents genuinely had nothing to do because + the producer was pre-aggregating. +- **Per-baseline effect** (identical across raw and sketch) — + right conclusion. The bottleneck was upstream of the pipeline. + +The mislabel is in the _remaining_ candidates: OTLP SDK +backpressure, Docker userland-proxy, kernel socket buffers. +None of these are the cause. All three would manifest as +rate-sensitive degradation; the new rate-invariance test rules +all three out simultaneously. + +## Implications for the paper + +The §6.2 "M× bandwidth reduction (raw vs sketch)" claim is more +fragile than it looks. With the current fake-exporter the raw +baselines (B0a / B0b / B1) are **already SDK pre-aggregated** +at 2000 pts/s — they are not emitting raw samples. The +bandwidth difference between raw and sketch baselines in the +existing CSVs therefore measures: + +- payload shape per exported data point (raw value vs + serialized sketch), + +**not** the paper's intended story of "raw emits every sample, +sketch emits one summary per window". + +Concretely, in `sweep-N1-20260422.csv`: + +- `b3-delta` @ N=1: `agent_in_kib_per_s = 16,462`, i.e. the + sketch processor receives 16 MiB/s of pre-aggregated input + from the SDK. +- `b2-full` @ N=1: `agent_out_kib_per_s = 167,701`, i.e. after + the sketch processor serializes a full sketch per window. + +Both are off the "truly raw sample stream" story that the +paper wants to tell. The "raw" side is not emitting every +sample — it's emitting a pre-aggregated sum per attribute set +per second. + +## Options + +Pick one before re-running any §6.2 sweep. Each has paper +implications. + +### A. Revert PR #182's interval change, accept the old number + +```go +reader := sdkmetric.NewPeriodicReader(exp, + sdkmetric.WithInterval(time.Second/time.Duration(rate))) +``` + +- Pros: Brings back the 325 k N=1 number, restores the + rate-sensitive throughput axis. +- Cons: At rate=1000 the SDK still pre-aggregates within each + 1 ms tick, but tick is small enough that each accumulated + `Counter.Add` typically fires just once per tick per + attribute set → emits ~1 data point per attribute set per + ms → approximates one exported point per `Add`. Works for + the paper but only accidentally. The trace-replay code path + in PR #182 still wants the fixed 1 s interval, so this + revert is a synthetic-only fix. + +### B. Dual-interval: synthetic keeps `1s/rate`, trace-replay keeps `1s` + +Guard with an `if traceFile != "" { … } else { … }` in +`main()`. Minimal behavioural change vs. pre-#182; restores +synthetic's rate-proportional emit; leaves trace replay +operating on its own tick. + +- Pros: All existing N=1 baseline CSVs become reproducible + again. Trace replay (the PR #182 feature) unaffected. +- Cons: Two code paths to maintain. Paper story still has the + structural issue in (C) below when cardinality grows. + +### C. Abandon OTel SDK aggregation for the "raw" baseline + +The cleanest paper framing is: "B0 / B1 emit every sample; +B2 / B3 emit one summary per window." The OTel SDK does not +give us "every sample" for Counters or Gauges — the SDK +fundamentally aggregates. Options: + +1. Emit raw samples via a plain OTLP gRPC client + (`otlpmetricgrpc` under the hood), bypassing the SDK + `MeterProvider` for the raw baseline only. +2. Use an OTel `Exponential Histogram` or per-sample + counter-with-exemplars path; exemplars survive SDK + aggregation. +3. Write samples as OTel _logs_ with structured fields and + treat the raw baseline as a log-shipping workload. (Honest + to production behaviour; breaks PromQL compatibility.) + +- Pros: Paper story matches code. "M× reduction" claim is + defensible. +- Cons: Bigger code change. Raw-baseline bandwidth numbers + will shift materially — likely up, maybe toward the + paper-motivating direction. + +### D. Accept current behaviour, rewrite the paper framing + +Reframe B0 / B1 as "SDK-aggregated raw" (one data point per +attribute set per scrape interval). This matches real-world +Prometheus behaviour (scrape at 15 s → one data point per +series per 15 s), so it is defensible. + +- Pros: No code change. Matches production patterns. +- Cons: "M× reduction" claim loses most of its force because + both sides are already aggregated; bandwidth difference + reduces to "raw value + labels" vs "sketch blob". Need to + frame §6.2 as "bytes per aggregated sample", not "throughput + reduction". Changes the §6.7 N-scale story: per-agent rate + is fundamentally `cardinality × (1 / scrape_interval)`, not + load-driven. + +## Recommendation + +**Option B + Option D combined**: + +- Land B as a fake-exporter code change (dual-interval) so the + pre-#182 N=1 sweep CSVs are reproducible for backward + comparison. +- Adopt D for the paper framing. Measure **bytes per window** + rather than points/s. The §6.2 story becomes "compressed + aggregated bytes" (sketch) vs "raw aggregated bytes" (B0 / + B1), which matches the real Prometheus pattern. +- Do not pursue C right now — the code churn isn't worth the + paper-claim upgrade if D is acceptable, and C needs to be + weighed against the reviewer-facing "why is your raw baseline + not the standard Prometheus scrape baseline". + +This means the remaining `nan` columns in +`sweep-N10-20260422.csv` still need filling (instrumentation +gap, independent of this RCA), but the "N=10 throughput +collapse" line in `DataCollector/TODO.md §1` can be closed. + +## Follow-up actions + +1. ~~Diagnose with variant sweeps (host network, SDK knobs, + kernel buffers)~~ — skipped; rate-invariance test proved + none of those are the cause. +2. Decide on Options A–D above with the user. +3. Depending on decision: implement B, run one N=1 + one N=10 + sweep to re-establish baselines. +4. Update `DataCollector/TODO.md §1` — close the "N=10 + throughput collapse" bullet, replace with "decide §6.2 + framing" bullet referencing this doc. +5. Update `paper-outline.md` experiment table — "throughput" + claim becomes "bandwidth (bytes/window)" if going with D. diff --git a/docs/paper-outline.md b/docs/paper-outline.md index 03859d54..eb37b202 100644 --- a/docs/paper-outline.md +++ b/docs/paper-outline.md @@ -129,13 +129,22 @@ workload + SLAs. cost model, replanning triggers 6. **Evaluation** - 6.1 Setup (workloads, baselines, deployment) - - 6.2 End-to-end benefits: B0 vs B3 on CPU, memory, - bandwidth, latency - - 6.3 Ablation: B1 vs B3 (no sketches), B2 vs B3 (no - controller) - - 6.4 Accuracy vs resource tradeoff: ε sweep, Pareto - - 6.5 Workload evolution: reconfig frequency × benefit - - 6.6 Failure modes: controller / agent / network partition + - 6.2 SDK-side three-axis ablation (see below + the + authoritative [`sdk-aggregation-three-axis-design.md`](sdk-aggregation-three-axis-design.md)). + Sub-sweeps 6.2a / 6.2b / 6.2c / 6.2d decompose the + bandwidth-reduction claim into its three independent + factors: time window `W`, label projection `L`, + encoding `agg_type`. + - 6.3 Cross-layer placement: same `agg_type` at SDK + vs agent vs backend, CPU / mem / bw tradeoff + - 6.4 Accuracy vs resource Pareto: ε sweep at fixed + `(W, L, agg_type)` operating point, Pareto curve + - 6.5 Planner quality: given query sets `Q_1,…,Q_k`, + does the controller's `(W, L, agg_type)` output match + hand-tuned ground truth? Independent of SDK emit cost. + - 6.6 Workload evolution: online replan latency after + injected drift; controller-in-loop end-to-end + - 6.7 Failure modes: controller / agent / network partition 7. **Related Work** — sketch DBs (Druid approximate, Pyramid, Moment-based), observability (Prometheus, VictoriaMetrics, M3, Thanos, Mimir), cross-tier query planning (ClickHouse @@ -149,14 +158,25 @@ workload + SLAs. | Paper claim | Experiment | Figure | |---|---|---| | "N% collector CPU reduction" | B1 vs B3 on Google cluster trace, 24h | stacked bar: CPU per node per baseline | -| "M× bandwidth reduction" | B1 vs B3 bandwidth over agent→backend link, mean + P99 | time-series of bytes/s + summary table | +| **"Bw reduction = time-factor × label-factor × encoding-factor"** | **6.2a / 6.2b / 6.2c (SDK-side three-axis ablation, each axis swept independently)** | **3 curves, each a mean + P99 band** | +| "End-to-end bw reduction on realistic queries" | **6.2d** — best `(W, L, agg_type)` per metric under `Q` vs `raw-buffer` at full label set + `W=15s` | Single stacked-bar: product of three factors | | "Query P99 latency: PromQL native vs sketch-answered" | B0 vs B3 on realistic query replay | latency CDF | -| "ε accuracy at M× resource savings" | Accuracy sweep at fixed workload; Pareto curve | accuracy vs cost Pareto scatter | +| "ε accuracy at M× resource savings" | Accuracy sweep at fixed `(W, L, agg_type)` operating point | accuracy vs cost Pareto scatter | +| "Cross-layer placement doesn't matter for correctness, but CPU/mem tradeoff differs" | **6.3** — same `agg_type` at SDK vs agent vs backend | stacked CPU/mem per layer | +| **"Planner's `(W, L, agg_type)` choice matches hand-tuned ideal within X%"** | **6.5** — offline planner vs ground truth over synthetic `Q` sets | match-rate curve | | "Controller responds to workload drift in T seconds" | Replan latency after injected drift | time-series with event markers | | "Cold S3 fallback adds Delta` (×5): +- Semantics: keep last emitted sketch bytes per reduced-attribute-key; + on tick, diff against current sketch and emit delta. If `L` or + sketch params changed since last tick, emit full sketch (not a + delta) and reset the reference. +- Encoding: byte-level XOR + zstd at first (simple, works for all + 5 sketch types with one codepath). Semantic delta (e.g., CMS cell + changes, DDSketch bucket changes) is a possible paper follow-up. +- Expected size: ~100 LOC each × 5 = 500 LOC + tests. + +Hot-reload of `L` at runtime: +- OTel Go SDK doesn't currently support replacing a View's + `AttributeFilter` after `MeterProvider` construction. For the + static §6 sweeps this is fine — each run is a fresh process. +- For the controller-in-loop §6.5 "planner pushes `L` change mid-run" + scenario, the SDK needs a hot-reload hook. This is tracked as a + separate implementation item; **not** a §6.2 blocker. + +## Paper §6 mapping (three-axis ablation) + +§6.2 splits into four sub-sweeps, each sweeping one axis while +holding the other two fixed at a representative operating point: + +| Sub-sweep | Fixed | Swept | Claim | +|---|---|---|---| +| **6.2a Time axis** | `L = full`, `agg = dd-full` | `W ∈ {1s, 15s, 60s, 300s}` | "Longer windows reduce bw / weaken freshness SLA" | +| **6.2b Label axis** | `W = 60s`, `agg = dd-full` | `\|L\| ∈ {0, 1, 2, 3, 4} dims` | "Projecting compatible label dims reduces bw by `orig/reduced`" | +| **6.2c Encoding axis** | `W = 60s`, `L = typical projection` | `agg ∈ {raw-buffer, dd-full, dd-delta, kll-full, kll-delta}` | "Sketch vs raw reduces per-point bytes; delta reduces further" | +| **6.2d End-to-end** | — | best `(W, L, agg)` per metric chosen by planner, vs `raw-buffer` at full label set + `W = 15s` | "Total bw reduction = time-factor × label-factor × encoding-factor" | + +§6.5 becomes a **planner-quality** experiment independent of the +SDK emit cost: given query sets `Q_1, …, Q_k`, inspect the +planner's `(W, L, agg)` output and compare against hand-tuned +ground truth. + +## Non-goals of this doc + +- How the agent collector further aggregates across SDK windows + — covered by `delta-transmission-design.md` and + `serf-compression-architecture.md`. +- Cost-model formulation — covered by + `controller-optimization-problem.md`. +- Query-side algebra and the algebra → physical-plan rewrite + rules — covered by `sketch-algebra-query-mapping.md` and + `controller/docs/query-to-sketch-translation.md`. + +## Implementation order (follow-up PRs) + +1. `AggregationRawBuffer` + unit tests + `Aggregation` enum wire-up. +2. ~~`AggregationDeltaSketch` ×5~~ — already present as + `DeltaTransmission: true` on the four sparse-state sketches + (DDSketch / CountSketch / CountMinSketch / HLLSketch). + Only `kll-delta` is a follow-up, and it's optional for §6.2. +3. `fake-exporter` knobs: drop `EXPORTER_RATE`; add + `EXPORTER_SDK_WINDOW`, `EXPORTER_SDK_PROJECTION`, + `EXPORTER_SDK_AGG`. Widen the synthetic label schema from + `{zone, pod}` (2 dims) to `{zone, rack, node, pod}` (4 dims) + so the `L`-axis sweep has range. +4. `measure-baseline.py`: add producer-side `producer_cpu_cores`, + `producer_rss_mib`, `producer_bytes_out_per_s` columns. +5. Run the four §6.2 sub-sweeps at `N = 1` on the 40-core dev + box. Produce four CSVs + four figures. +6. (Separate track) OpAMP hot-reload of `L` for §6.5.