ProjectASAP · zzylol · Apr 23, 2026 · Apr 23, 2026 · Apr 23, 2026 · Apr 23, 2026
diff --git a/PROGRESS.md b/PROGRESS.md
@@ -1,6 +1,14 @@
 # DataCollector — Implementation Progress
 
-_Last updated: 2026-03-14 (SDK pre-aggregation update)_
+_Last updated: 2026-04-23 — three-axis SDK framework formalized_
+
+See [`docs/sdk-aggregation-three-axis-design.md`](docs/sdk-aggregation-three-axis-design.md)
+for the current authoritative design of the SDK decision point
+(time window `W` × label projection `L` × encoding `agg_type`).
+The 2026-03-14 SDK pre-aggregation batch below covers the
+`*-full` encoding column; the `raw-buffer` and `*-delta`
+columns are still open (see "Outstanding SDK aggregators"
+section at the bottom of this file).
 
 ---
 
@@ -227,3 +235,54 @@ OpenTelemetry Collector (custom build via OCB)
 Prometheus / Grafana
   Metric: <name>_hll_cardinality{host="...", metric="..."} <estimate>
 ```
+
+---
+
+## 2026-04-23 update — three-axis SDK framework
+
+The five SDK pre-aggregation aggregators above (DDSketch / KLL /
+CountSketch / CountMinSketch / HLLSketch) all implement the
+`*-full` encoding slot of the three-axis `(W, L, agg_type)`
+framework defined in
+[`docs/sdk-aggregation-three-axis-design.md`](docs/sdk-aggregation-three-axis-design.md).
+
+### Outstanding SDK aggregators (P1 for paper §6.2)
+
+Correction after a read of
+`opentelemetry-go-patch/sdk/metric/aggregation.go` — **delta
+encoding is already a flag on the four sparse-state sketches**
+(`DeltaTransmission: true`), not a separate aggregator. So the
+real gap is smaller than the earlier plan:
+
+| Aggregator | Slot | Status | Notes |
+|---|---|---|---|
+| `AggregationRawBuffer` | `agg_type=raw-buffer` | ❌ | Buffers `(ts, attrs, value)` tuples within `W`, emits batch of `NumberDataPoint`s per tick. Overflow: drop + drop-counter metric. ~150 LOC. |
+| `AggregationKLLSketch.DeltaTransmission` | `agg_type=kll-delta` | ❌ | KLL's multi-level sample buffers don't support a natural byte-diff; adding delta requires exposing per-level internals from `sketchlib-go` or shipping incremental adds. **Not a §6.2 blocker** (see design doc for rationale). |
+
+The other four sketch delta slots (DDSketch / CountSketch /
+CountMinSketch / HLLSketch) already work via the
+`DeltaTransmission: true` flag from the 2026-03-14 batch above.
+
+### Outstanding SDK runtime support
+
+- **Hot-reload of View `AttributeFilter`** — required for the
+  controller-in-loop §6.5 scenario where the planner pushes a
+  new projection `L` mid-run. Upstream OTel Go SDK doesn't
+  support replacing a View's filter after MeterProvider
+  construction; needs a small patch in
+  `opentelemetry-go-patch/sdk/metric/` to expose a swap API.
+  Not a §6.2 blocker (each static sweep run is a fresh
+  process).
+
+### Downstream dependents
+
+- `deploy/fake-exporter/main.go` — needs to drop
+  `EXPORTER_RATE` (semantically meaningless now — see
+  [`docs/n10-bottleneck-rca.md`](docs/n10-bottleneck-rca.md))
+  and expose `EXPORTER_SDK_WINDOW`, `EXPORTER_SDK_PROJECTION`,
+  `EXPORTER_SDK_AGG`. Widen the synthetic label schema from 2
+  dims (`{zone, pod}`) to 4 dims (`{zone, rack, node, pod}`)
+  so the `L`-axis sweep has range.
+- `deploy/scripts/measure-baseline.py` — add producer-side
+  columns (`producer_cpu_cores`, `producer_rss_mib`,
+  `producer_bytes_out_per_s`).
diff --git a/TODO.md b/TODO.md
@@ -1,6 +1,6 @@
 # TODO — DataCollector + controller for paper submission
 
-_Last updated: 2026-04-23 (post N=10 sweep, PRs #168–#185)_
+_Last updated: 2026-04-23 — N=10 "collapse" diagnosed + new 3-axis SDK design landed_
 
 This doc lists what's left to get a VLDB / SIGMOD submission out
 the door. For the v1 paper we keep updating `controller/`
@@ -43,35 +43,54 @@ landed over PRs #168–#185. Briefly:
 
 ## For paper submission (blocker)
 
-### 1. Multi-agent scale — N=10 throughput collapse (P0)
-
-N=10 sweep (#185, 2026-04-22) surfaced a system-wide bottleneck:
-per-agent throughput collapses from 130k–326k pts/s at N=1 to a
-universal ~2k floor at N=10, identical across all six baselines
-(so it's a topology / producer-SDK issue, not a sketch pipeline
-issue). Agents are near-idle (0.01c, 220 MiB). Gateway sees
-10 × 2k = 20k total, i.e. coordinated throttling.
-
-Until this is diagnosed and fixed, the paper's "scales linearly
-N ∈ {1, 10, 100}" claim is unsupported.
-
-Candidate causes, ranked:
-
-1. **fake-exporter OTLP/gRPC SDK backpressure.** 10 producers ×
-   1M nominal points/s (rate × cardinality) likely overwhelms a
-   shared socket or SDK reader limit. First thing to try: tune
-   `WithMaxQueueSize` / `WithMaxExportBatchSize`, or drop to a
-   raw gRPC client for the scale sweep.
-2. **Docker userland-proxy CPU contention** across many
-   simultaneous gRPC connections on the default bridge network.
-   Cheap test: switch to `network_mode: host` for agents +
-   gateway and re-run.
-3. **Kernel socket buffers.** `net.core.somaxconn /
-   net.core.rmem_max` at stock Ubuntu defaults may cap inbound
-   gRPC streams.
-
-Post-fix deliverable: re-run N=1, N=10, N=100 sweeps; paper's
-§6.7 "scales linearly" figure uses those three points.
+### 1. SDK three-axis aggregation framework (P0) — supersedes the "N=10 throughput collapse" blocker
+
+The N=10 "collapse" (#185) turned out not to be a bottleneck —
+[`docs/n10-bottleneck-rca.md`](docs/n10-bottleneck-rca.md) walks
+through the diagnosis. The 2 k pts/s floor was the OTel SDK's
+correct pre-aggregation output at `interval=1 s, cardinality=1000,
+2 instruments`, independent of input rate. That finding reframed
+the paper's §6.2 bandwidth claim as a **three-independent-factor
+product** (see
+[`docs/sdk-aggregation-three-axis-design.md`](docs/sdk-aggregation-three-axis-design.md)).
+
+Concrete work items (P0 because §6.2 can't run without them):
+
+- [ ] **`AggregationRawBuffer`** in
+      `opentelemetry-go-patch/sdk/metric/aggregation.go` —
+      `(ts, attrs, value)` buffer, emit batch per tick, drop +
+      drop-counter on overflow. ~150 LOC + tests.
+- [x] ~~`Aggregation<X>Delta` × 5~~ — on inspection, four of five
+      (DDSketch / CS / CMS / HLL) already have `DeltaTransmission`
+      as a flag on the `*-full` aggregator (2026-03-14 batch).
+      Only `kll-delta` is missing and is not a §6.2 blocker
+      (KLL's multi-level buffer structure needs a different
+      delta strategy — see
+      [`docs/sdk-aggregation-three-axis-design.md`](docs/sdk-aggregation-three-axis-design.md)).
+- [ ] **`fake-exporter` rewrite** (`deploy/fake-exporter/main.go`)
+      — drop `EXPORTER_RATE`; add `EXPORTER_SDK_WINDOW`,
+      `EXPORTER_SDK_PROJECTION`, `EXPORTER_SDK_AGG`. Widen label
+      schema from 2 dims to 4 (`{zone, rack, node, pod}`).
+- [ ] **`measure-baseline.py`** producer-side columns —
+      `producer_cpu_cores`, `producer_rss_mib`,
+      `producer_bytes_out_per_s` (scrape the fake-exporter
+      container's cgroup + interface counters).
+- [ ] **§6.2 sweeps** at `N=1`:
+      - 6.2a time: `W ∈ {1s, 15s, 60s, 300s}` ×
+        `L=full, agg=dd-full`
+      - 6.2b label: `\|L\| ∈ {0,1,2,3,4}` × `W=60s, agg=dd-full`
+      - 6.2c encoding: `agg ∈ {raw-buffer, dd-full, dd-delta,
+        kll-full, kll-delta, cms-full, hll-full}` ×
+        `W=60s, L=typical projection`
+      - 6.2d combined: best per-metric triple vs `raw-buffer +
+        full-L + W=15s`.
+- [ ] **N-scale sweep rerun** at fixed representative
+      `(W=60s, L=subset, agg=dd-delta)` across `N ∈ {1, 10, 100}`.
+      This is now the honest scalability test — the 2 k floor
+      from #185 is expected; we're looking for whether gateway /
+      backend hold up as aggregate ingress grows.
+
+Depends on nothing upstream; can start immediately.
 
 ### 2. Instrumentation — fill the `nan` columns (P1)
 

diff --git a/deploy/TODO.md b/deploy/TODO.md
@@ -26,9 +26,29 @@ is instrumentation completeness, Helm templates, and polish.
 
 ## Instrumentation gaps (P1 — blocks §6.2/6.3 figures)
 
-The N=10 sweep CSV has several `nan` cells that need filling
-before the paper plots can be drawn. Tracked top-level as
-`DataCollector/TODO.md §2`; implementation lives here.
+The §6.2 sub-sweeps defined in
+[`../docs/sdk-aggregation-three-axis-design.md`](../docs/sdk-aggregation-three-axis-design.md)
+require **producer-side** measurements that don't exist yet;
+the N=10 sweep CSV also has several `nan` cells on the
+collector side. Both tracked top-level as
+`DataCollector/TODO.md §1-ish` (the old §2 wording is rolled
+into §1).
+
+### Producer-side columns (new, P0)
+
+- [ ] **`producer_cpu_cores`** — `docker stats` or cgroup
+      read on the `fake-exporter` container. The §6.2
+      SDK-side CPU claim is "how much does `agg_type=*-delta`
+      cost the producer vs `*-full`".
+- [ ] **`producer_rss_mib`** — same source. `agg_type=raw-buffer`
+      is expected to have the largest producer RSS (sample
+      buffer); we need to measure the knee vs window `W`.
+- [ ] **`producer_bytes_out_per_s`** — agent-container
+      `container_network_transmit_bytes_total{name="fake-exporter"}`,
+      `rate()` over the measurement window. The main §6.2
+      bandwidth axis; we've been inferring this from
+      gateway-side counters which conflates multiple agents
+      at N > 1.
 
 - [ ] **Byte counters on raw and Gorilla baselines.**
       `agent_in_kib_per_s` / `agent_out_kib_per_s` are `nan`

diff --git a/deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv b/deploy/eval-results/n10-diagnosis/rate-invariance-20260423.csv
@@ -0,0 +1,3 @@
+baseline,scale,rate,cardinality,agent_cpu_cores,agent_rss_mib,agent_in_kib_per_s,agent_out_kib_per_s,agent_points_per_s,gateway_cpu_cores,gateway_rss_mib,gateway_points_per_s,gateway_out_series_per_s,backend_cpu_pct,backend_rss_mib,backend_samples_per_s,backend_query_p99_ms
+b0a-raw-stream,N1,1000,1000,nan,nan,nan,nan,nan,0.043,196.652,2000.000,2000.000,nan,nan,2001.000,nan
+b0a-rate10k,N1,10000,1000,nan,nan,nan,nan,nan,0.044,196.422,2000.000,2000.000,1.690,11.770,2001.000,nan