feat: GOS unified edge telemetry — per-cell sketch pipeline, SDK-side NitroSketch row admission, F2 removal by zzylol · Pull Request #515 · ProjectASAP/ASAPCollector

zzylol · 2026-07-16T15:39:09Z

Summary

Supersedes the closed #507/#508/#511–#514 stack (all verified as strict
subsets of this branch's tree — see closing comments on those PRs). This is
the consolidated GOS (Geometric-OctoSketch) unified edge-telemetry framework:
per-cell sketch precision replacing whole-sketch monitoring, SDK-side
NitroSketch-style row-admission sampling replacing collector-side sampling,
and a large evaluation pass validating both against real datasets (DEBS 2022,
Google cluster trace).

Canonical design doc: docs/design-gos-unified-edge-telemetry.md.
Sampling derivation: docs/sampling-cdm-gos-derivations.md.
Eval results: docs/gos-eval-results.md, docs/phase-2.md.

What's actually live (net of two mid-branch pivots)

Two major decisions were made and executed within this branch — reviewers
should judge the final state, not the intermediate commits that were
later reverted:

F2 / geometric-CDM monitoring was added, then fully removed. The
per-cell sketch pipeline ships CMS/CountSketch/DDSketch cells to the
backend, which reconstructs the full sketch and answers any query — F2
(‖f‖₂²) is just one such query, so the dedicated geometric safe-zone F2
monitor (a bandwidth-saving gate) was redundant once cells ship anyway.
Removed cleanly from both this repo and ASAPQuery-backend (per-cell
sketches had zero dependency on F2Engine); monitorpb/asap_otel_proto
regenerated to drop MonitorReport.sketch/RefBroadcast/CoordToEdge.ref.
Sampling/ε-floor (epsilon_sample_floor, sampling_margin, allocate_p)
is kept — separate axis, unaffected.
Collector-side ConsistentSampler/otlpfilter sampling was added, then
replaced by SDK-side NitroSketch row admission. The SDK now decides, per
raw occurrence at Record() time, which rows of the target physical
sketch (AggregationIdentity{AggID, Filter} — one instance per
AggregateBy group, matching NitroSketch's one-update-stream-per-sketch
model) get admitted, using a GeometricSampler fed by a live
sample_p grant from the coordinator. The admission bitmask ships over
OTLP (new AggregationRowSampledSketch metricdata type; wire-encoded as a
standard Gauge/NumberDataPoint with 3 reserved attributes — no new proto
message needed) and the collector applies it verbatim via
Sketch.ApplyAdmittedOccurrence, never re-deriving its own sampling
decision. AggID was also fixed to be the real content-addressed
PolicyFingerprint (matching ASAPQuery-backend's Rust implementation
byte-for-byte) instead of a bug that hashed only the metric name.

Key components

Per-cell delta pipeline (GOS Phase 2): anisotropic/isotropic per-cell
delta threshold at the edge, sub-window emit wiring, sparse-delta
broadcast application — sketchlib-go's ComputeDeltaPerCell (merge
sketchlib-go#69 first).
SDK-side row-admission sampling: PolicyFingerprint,
AggregationIdentity/AggregationRouter, liveSampleGrant (mirrors
otel-app/sample_controller.go), AggregationRowSampledSketch aggregation
type, OTLP wire transport (both otlpmetricgrpc/otlpmetrichttp, unified
from one backfilled template — a prior drift meant otlpmetrichttp was
silently missing SeriesId wiring that otlpmetricgrpc had; fixed here
too), and collector-side decode in asapedgeprocessor.
Removed: the 5 sketch merge processors + nopprocessor (retired
gateway hop), AllocateSampleRates, ConsistentSampler, otlpfilter,
F2Engine and all its wiring/docs.
Docs: consolidated/slimmed — dormant integration docs relocated,
phase-2 history docs merged, canonical-pointer banners added where content
moved.
Eval: DEBS 2022 real-backend query-accuracy harness (all 4 sketch query
types), integrated ε-sweep (accuracy·latency·freshness·resources,
single-node + cluster), C1/C4 bandwidth and accuracy experiments,
anisotropic-vs-isotropic per-cell delta measurement, raw-vs-sketch
cardinality crossover.

Test plan

Each commit was built/tested at the time it landed (per this repo's
convention — see individual commit messages).
This session's final increment (AggID fix, AggregationRowSampledSketch,
OTLP wire transport for both exporters, asapedgeprocessor decode) was
independently build/vet/test verified across
asap-precompute-go, opentelemetry-go's SDK + both OTLP exporters,
and opentelemetry-collector-contrib's asapedgeprocessor, including
new end-to-end tests for the row-sampled admission path (rescale-by-p
correctness, reserved-attribute stripping from series identity,
defensive drop on SDK/collector AggID disagreement).
CI on this PR (full matrix).
sketchlib-go#69 should merge first (per-cell delta primitives this
branch depends on).

🤖 Generated with Claude Code

…/wire in one run Extends pareto_sweep.py: per arm also measures query latency (perf_counter p50/p99), data freshness (emit→queryable poll from producer end), and per-process RSS, on top of the existing accuracy + cpu + wire. Sweeps the admission p (= the ε-floor the autonomous coordinator sets) and reports the implied ε. Phase-1 single-node table + honest caveats (accuracy ~13% calibration; freshness ~19ms is a compressed-replay artifact — real freshness/per-component/cold-tier = Phase-2 cluster). See INTEGRATED_SWEEP_RESULTS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…onent + cold-tier Thread -warm-sample-p through run_demo.sh producers (default 1.0, backward-compat). Add epsilon_cluster_sweep.sh: per admission p bring up the warm+cold asap stack (wall-clock paced) and capture per-tier freshness, per-container CPU/mem, per-node NIC bandwidth, and warm query latency. Baseline p=1.0 result in PHASE2_INTEGRATED_RESULTS.md: real freshness warm ~1.0s/archive ~0.66s (the column single-node returned None for), and the per-component split (warm sketch backend = 2% CPU / 117MiB for 5000+ sketches; cost at edge + cold archiver). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… invariant to p 5-arm cluster sweep (p 1.0→0.05). Clean result: query latency (~1.1ms p50) and warm sketch-backend cost (~1% CPU / 68MiB) are FLAT across the full sampling range — the ε-floor keeps one sketch per series so the serving path is p-independent. Freshness/edge-CPU/NIC show no systematic p-trend (expected: window-seal latency and producer-side work are independent of admission sampling); per-arm scatter is short-soak + multi-series probe noise, baseline run is the reliable freshness point. Adds aggregate_phase2_sweep.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…99, robust to ε Thread trace-replay (OTELAPP_TRACE_MOUNT/OTELAPP_TRACE_ARGS) through run_demo.sh producers so the real google-cluster-2019 cpu_rate trace flows through the distributed stack with -warm-sample-p admission sampling applied. epsilon_accuracy_sweep.sh sweeps p and queries the warm DDSketch via quantile_over_time(q,metric[5m]) (the correct form; histogram_quantile routes to thanos). accuracy=1-|sketch-GT|/GT vs exact offline GT (p99=0.043274). Result: p99 accuracy ~90% flat across p 1.0->0.05 (ε 0->0.056) — sampling is accuracy-robust; cluster DDSketch ~1-8% err vs single-node's 13% (better-calibrated window). Completes the integrated table: accuracy+latency+freshness+per-component, one stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Cold query explicitly tested (not inferred): count_over_time[10m] -> thanos_query -> 24957/25395 archived samples; timestamp-consistency via window-scaling (1m/2m empty matching the 2m+1m archive lag, 5m=16.7k, 10m=25k), cross-producer agreement ~3%, value distribution matches GT (warm p99 0.042-0.049 vs 0.0433, archived values in trace domain). Honest limit recorded: cold values confirmed indirectly (value funcs route to warm frontend). gorilla-merger: RestartCount=0 OOMKilled=false, 21MiB/32GiB, no OOM; MinIO S3 upload not observed completing in-window. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Earlier total_objs=0 was a pre-compaction-cycle reading, not a gap. Watching across the 5m compactor cycle: 'shipper uploaded blocks uploaded=1' to bucket=asap-gorilla-tsdb, MinIO holds a real TSDB block (chunks/index/meta.json, erasure-coded). Cold tier durable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Lets run_demo producers run as coordinated CDM edges (-coordinator-url / -monitor-config-url / -monitor-key) via env, alongside the existing OTELAPP_WARM_SAMPLE_P / OTELAPP_TRACE_ARGS knobs. Used to validate the autonomous /plan/auto → coordinator → ε-floor-p loop end-to-end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

… fix retired √(f/rate) ref Consolidated comparative-rigor plan: 5 claims + Pareto + autonomous-allocation + ablations, each crossed with real datasets (Google-cluster-2019 staged, DEBS-2022 downloading, 3rd gap) and real baselines (raw+codec, Prometheus/Thanos/VM, NitroSketch/OmniSketch, Cormode-CDM, ASAP ablations). Priority gaps: real baselines, statistical rigor (trials+CI), all-6-family accuracy, DEBS e2e, autonomous-alloc quality vs oracle, scale. Also corrected the headline's stale per-key sampling law to the unified whole-sketch ε-floor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…g not engaged, KLL/CMS) Fresh run on main: DDSketch p50 0.72% + HLL 0.33% solid (natural-fit gauge families); KLL p50 regressed 0.0007→0.0597, CMS one-sided violated (143<244), agent shipped 57MB raw not ~1MB sketched. Flags gate C1/C4. Data-fitness finding: frequency families belong on DEBS (symbol-trade freq), not gct gauge data. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…perating-point) 57MB wire = per-family slices all contain 8 metric aliases (agent sketches 1, forwards 7 raw); accuracy is valid. KLL median = small-N rank coarseness (DDSketch buckets win small-N); CMS one-sided = gauge data has no count to over-estimate (frequency families → DEBS). C1 wire to be re-measured with true family slices. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…C1 clean-wire harness fix identified C4 cell ✅ for the natural-fit gauge families (DDSketch p50 0.72%, HLL 0.33%) after root-cause; C1 cell notes the make_perfamily un-slice + ship-wait coupling that blocks clean per-family wire. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…→ clean sketch-vs-raw wire Fixes the two gaps that polluted per-family wire: (1) c1_wire.py slices the data to {cpu_rate Sum-anchor, memory_usage, ONE family metric} (drops the 6 noise aliases that forwarded raw → 57MB), (2) agent-raw-coldoff.yaml (no asap_edge) replays the same slice as the raw baseline. Captures W_sketch vs W_raw over N trials with 95% CI. Validated (ddsketch, real gct, 1 trial): W_sketch 0.99MB vs W_raw 33.7MB = 34.1x reduction, accuracy intact (p50 0.72%). Confirms the prior 57MB was an un-sliced-data artifact, not a regression (W_sketch=0.99MB matches the prior good run). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…(real gct, 3 trials+CI) Matrix C1 cell → measured ✅. Fixed harness (true family slice + raw-forward arm) gives clean sketch-vs-raw wire with tight CIs; W_sketch reproduces the prior committed numbers exactly, settling the '57MB' question (un-sliced data, not a regression). Compression/Prom baselines remain the open comparative gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…2.7x over raw+gzip Compression baseline (b0-gzip/zstd): gzip on the actual raw payload = 12.3x encoding factor; ASAP's 33.8x decomposes as encoding(12.3x) x residual-aggregation(2.7x) — internally consistent (12.3*2.7≈33.8). Sketching still beats a gzip-compressed raw baseline by ~2.7x. Honest caveat: --network host single-node can't isolate the agent→backend wire (loopback method failed, all arms ~35MB); clean on-wire compression belongs on the cluster NIC. Configs+driver ready for it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…60x smaller; X1 ✅ Updated Fig 9 to the unified ε-floor (retired √(f/rate)); added the standalone algorithm result (real DEBS rates, equal bandwidth): ε-floor max rel-err 0.06-0.41 vs fixed-p 3.9-40.9. First real baseline vs academic prior art (NitroSketch). Matrix X1 → ✅. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…hole-sketch findings Retract the 40-60x per-key claim (circular oracle f_k + retired law). Corrected: whole-sketch ε-floor on real CMS — 17x insert-throughput at bounded L2, memory constant, p derived from ε+observable R. Honest: ε-floor bounds L2 not per-key (rare-key accuracy inherently limited under sampling); right metric = L2/heavy-hitter (TODO). Fleet modest (1.3x, 3 exchanges). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…t 1) asap-precompute-rs hardcoded sample_p=0.0 in all wrappers — it could decode a sampled envelope (query-side rescale) but never PRODUCE one, so a Rust edge under the CDM coordinator's ε-floor grant would ship unsampled full-rate sketches. Mirror Go's coordinated path (precompute.SampleSetter, applyGrantedSampleP): - New sampling.rs: GeometricSampler (NitroSketch geometric skip-sampling, splitmix64) + wire_sample_p (stamp 0.0 when exact for byte-parity, else p). - SampleSetter trait in precompute.rs — implemented by exactly CMS, CountSketch, DDSketch (the additive families). Sum/KLL/HLL deliberately excluded, matching Go's switch in applyGrantedSampleP (HLL keeps its own hash-threshold path). - Each of the 3 wrappers: sample_p field + GeometricSampler, with_sample_p builder + set_sample_p (reseed), admit-gate in update(), stamp wire_sample_p in the envelope, reseed on reset(). Seeds mirror Go (cms ZMPC / cs 0x5a3e06d / dd DDSP). - Count sketches rescale by 1/p on query; DDSketch is scale-invariant (samples to shed work, no rescale) — both still stamp p. Exact path unchanged: sample_p stays 0.0 → #243 byte-parity preserved (all 57+11+27 tests green). New test asserts admit≈p fraction + envelope stamp + SetSampleP. Part 2 (runtime hook: derive p from the engine's granted ε-floor and install the window sample-hook, mirroring Go's applyGrantedSampleP wiring) is a follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…antile (#30) Locks in the quantile case: sampled DDSketch admits ~p of updates (total_count ~p×) but its quantile shape survives (scale-invariant, no 1/p rescale), and the envelope stamps p while the exact path stamps 0.0 for byte-parity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…+ skewed fleet) F1-aggregate held at ε (0.050@ε=.05); point-query degrades as ε·√(R/f_k) (heavy survive, rare lost); 22× insert-tput; synthetic fleet cold-edge 7× better than fixed-p. Fig 9 → ✅ algorithm. X1 cell updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ollector_benchmark Relocate the standalone ε-floor benchmark INTO ASAPCollector (its own Go module, relative replace → ../../../sketchlib-go, mirroring cardinality_crossover) so the paper-eval code lives with the rest of the eval and sketchlib-go stays unmodified. Whole-sketch ε-floor p=1/(1+ε²R) on real CMS over real DEBS: - 20× insert throughput (183 vs 8.9 Mupd/s), CMS memory constant - accuracy LAW: F1-aggregate held at ≈ε (0.050@ε=.05); point query degrades as ε·√(R/f_k) (heavy survive, rare lost) - synthetic skewed fleet (rate-CV=2.9): ε-floor equalizes per-edge err at ε; fixed-p under-protects the cold edge 7× (0.148 vs 0.020) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Edge side of the F2 whole-sketch monitor and the unified GOS framework doc. - f2engine.go: F2Engine — local safe-zone test (mirrors Rust f2.rs), ship/silent gating, RefBroadcast handling; safe radius uses (1-ε)τ. - types.go: FunctionalF2, F2Mode, Sketch report payload, RefBroadcast directive. - grpcclient: sketch payload + RefBroadcast wiring; regenerated stubs. - countsketch.go: CellMatrix() accessor for the whole-sketch monitor. - cmd/f2driver: multi-edge eval driver; deploy/.../f2_monitor_eval.sh. - docs/design-gos-unified-edge-telemetry.md: the full GOS framework (error bound, water-filling thresholds, tunable mem/compute/comm objective, WZ optimality); docs/f2-geometric-monitoring.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The edge-side twin of the Rust control_plane threshold_alloc: compute the F2 GOS delta threshold from local sketch state instead of a fixed configured value. - gos_threshold.go: Go mirror of threshold_alloc — F2 isotropic closed form T = ε‖Ĉ‖/(2k√(dw)) + box-constrained water-filling AllocateThresholds (T_j ∝ √(V_j/|g_j|), sampling floor, query/freshness caps). - CountSketchWrapper.ComputeGosDelta: emits a per-cell delta using the norm- relative GOS threshold via the EXISTING sparse-delta path (which already gates each cell by |ΔS[r][c]| ≥ threshold) — no wire/serialization change. The threshold is adaptive (scales with ‖Ĉ‖) so whole-sketch relative error stays within ε as the sketch grows. The anisotropic (gradient-weighted) per-cell variant needs a vector-threshold delta in sketchlib-go (follow-up); the F2 isotropic case is uniform so the scalar path suffices. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ig-gated) Route the sub-window and full-window delta emit paths through the GOS F2 per-cell threshold when PrecomputeConfig.GosDeltaEpsilon > 0, else the fixed DeltaThreshold (default → unchanged behavior). - config.go: GosDeltaEpsilon + GosSites knobs. - countsketch.go: GosDeltaThreshold(ε, k) → ceil(ε‖Ĉ‖/(2k√(dw))). - precompute.go: gosDeltaThreshold() gate at both ComputeSubWindowDelta and ComputeDelta call sites (structural interface assert; no Sketch iface change). Backend apply_delta is unchanged — the wire format (sparse cells) is identical; only which cells cross the (now norm-relative) threshold differs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ry toggle) Make the per-cell threshold shape a config knob so the edge-memory vs communication tradeoff is tunable: - GosAnisotropic=false (default): ISOTROPIC scalar T=ε‖Ĉ‖/(2k√(dw)), O(1) mem. - GosAnisotropic=true: gradient-weighted per-cell {T_j} (water-filling T_j∝√(V_j/|g_j|), g_j=2|Ĉ_j|), O(d·w) mem, less communication on skewed data. - config.go: GosAnisotropic knob. - countsketch.go: SetGosMode + ComputeDeltaAgainst branch; computeAnisotropicDelta + gosThresholdMatrix (AllocateThresholds → sketchlib ComputeDeltaPerCell). Heap-msgpack path falls back to isotropic. - precompute.go: applyGosMode() sets the mode per emit (structural assert; no Sketch iface change); delta call passes the fixed threshold, wrapper overrides. DEPENDS ON sketchlib-go#69 (ComputeDeltaPerCell). Backend apply_delta unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

OnRef now handles both reference forms: a Full frame replaces the cached C_ref; a sparse delta (IsDelta) applies [rowIdx,colIdx,vals] cell-wise onto it. A delta with no cached base is dropped and counted (a Full keyframe follows). Wire codec is sketchlib-go asapmsgpack.UnmarshalCountSketchDeltaSparse (byte-parity with the Rust rmp_serde 5-tuple producer — verified). DEPENDS ON sketchlib-go#69 (sparse delta codec commit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…mpling Refine the sampling model to the SDK↔collector split (§3.1): the SDK does row-admission only — by NitroSketch geometric skip-sampling (O(1) RNG per admitted update, not d coins per item; whole-item skip is O(1)) — and never hashes; the agent collector computes the row hashes and updates counters ONLY for admitted rows, with 1/p inverse-probability weighting. A sample admitting no row is dropped at the source (collector skips its deserialization + hashing). Per-row knob p_{i,r}; unbiasedness + variance/ε_sa; cost model split into Comp_sdk (RNG) + Comp_coll (hash) both ∝ (Σ_r p_{i,r})·rate; wire volume × (1−∏(1−p)). This matches the existing sketchlib-go GeometricSampler. Ref: NitroSketch (Liu et al., SIGCOMM 2019). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ellites Verified the 3 satellites are COMPLEMENTARY, not duplicate: each carries a derivation the canonical doc omits (nitrosketch: the SDK->collector split survival proof + per-family applicability + empirical; tumbling-cost: the per-family two-disciplines cost tables; taxonomy: the full 2D grid + (agg_id,group_key) keying). So deleting would lose content -- instead unify the reader's entry point: each now banners sampling-cdm-gos-derivations.md as canonical and states what is unique here. Navigation consolidated, no content dropped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

phase-2-{execution-plan,perf-bench-go,perf-deployment}.md all record COMPLETED phase-2 work (runtime-extraction map, Go perf audit, perf deployment). Concatenate into a single phase-2.md history archive (each former doc demoted to a ## section with a provenance comment), delete the 3 originals, and repoint the ADR-0002 inbound link. 3 files → 1; content preserved verbatim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…es; archive resolved investigation eval-query-plan.md (0 inbound, the per-dataset query list) and eval-instrumentation-notes.md (the sweep-CSV column reference) are companions to the §6 figure plan -> merge both into evaluation-plan-figures.md as sections; repoint the runbook's inbound link. eval-label-axis-cpu-rootcause.md is a COMPLETED 2026-05 investigation -> move to docs/archive/ with a resolved banner. sdk-cost-evaluation.md kept (3 inbound, distinct topic). 5 eval docs -> 2 (+1 archived); content preserved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The *mergeprocessor Go processors (countsketch/countminsketch/ddsketch/hll/kll) were the gateway-side sketch accumulators. The asap-gateway hop was retired (#400) and cross-edge merge now runs on the backend data_plane (Rust); no pipeline ever wired them and nothing imports them (grep-confirmed across configs, code, and both manifests). They were dead weight compiled into every asap-otel binary. Remove their gomod+path entries from all OCB manifests (builder-config.yaml, builder-config-sketches.yaml, asap-otel-opamp), delete the 5 patch processor dirs, and note the removal in delta-transmission-design.md (design of record kept; mechanism now realized backend-side). Verified: OCB rebuild succeeds (exit 0) — the manifest resolves and the collector compiles without them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ale too small?') The default whole-sketch F2 workload has only H=4 distinct keys, so raw (~7KB) beats both sketch modes -- a fixed d*w~=11.5KB/ship cost cannot win at tiny cardinality. Replace the prose estimate with the measured F2_KEYS sweep: raw scales linearly with H while the sketch cost is H-independent, so the crossover is H~=500 (vs distributed) / ~280 (vs geometric); at H=32768 sketch wins 73x/127x. Clarify the two orthogonal axes: sketch-vs-raw is a cardinality question (crossover + C1-wire), geometric-vs-distributed is an H-independent monitoring question (ship-on-violation), so the 4.0x/1.74x headline holds at any scale; the H=4 run isolates the latter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…e-sketch broadcast limit Add tau=250000*H auto-scaling to f2_wholesketch_cluster.sh (keeps the ramp crossing at the same fractional step at any cardinality) and default it to H=2048 realistic distinct-key count, F2_KEYS threaded to the driver over ssh. Measured at H=2048 (recorded in *_h2048.csv), two honest findings: - Sketch beats raw at scale: ramp distributed 923,280 (fixed d*w) is 4.3x smaller than raw 4,007,440 — resolves the 'raw is cheapest' concern once H is realistic. - Geometric's ramp win does NOT survive a dense sketch: geometric ramp rose 531,132 (H=4) -> 1,647,966 (H=2048) and now LOSES to distributed. Egress decomposes to ~4*28*11,541 = near-full-matrix broadcasts: 2048 keys saturate the 1280-cell sketch, so sparse C_ref deltas degenerate to full matrices and the O(k) amplification returns (design sec 12 #1 + small-norm limit, now measured). Stable geometric still wins 4x (only 4 ships). The protocol pays off when the sketch is sparse relative to its cell budget. Corrects the earlier 'geometric ratio is H-independent' claim: the ship COUNT is ~H-independent, the BYTE count is not (broadcast grows with density). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…t result f2driver gains a 'zipf' ramp pattern (per-key drift weighted by Zipf(1.2), Σ weight = H·drift so τ stays comparable) to probe whether input skew helps the thresholded broadcast gate. It does not (Count-Sketch homogenizes cell magnitudes), which is now documented in gos-eval-results.md §2 alongside the density sweep: the gate is implemented, safe (H=4 byte-identical), and the right mechanism, but its Count-Sketch payoff is capped; the effective high-cardinality lever is sizing w to keep fill H/(d·w) ≪ 1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… workload) Add a real-data F2 path: debs_f2_trace.py bins the DEBS 2022 trading-day CSV (symbol = key) into step/edge per-symbol event counts, f2driver gains an F2_TRACE replay mode (sketch + raw), and f2_debs_eval.sh runs the raw / distributed / geometric matrix against the real trace. Measured (H=5493 real symbols, d=5 w=4096, tau=2.5e10, first 4M events; eval-8node/f2_debs.csv): raw 99,122,250 B (ship every event) distributed 14,747,280 B (6.7x less than raw) geometric 13,033,555 B (7.6x less than raw; beats distributed, 40/80 ships) all fire the alert at the real F2 crossing. On real data both claims hold by construction: sketch >> raw (real cardinality) and geometric > distributed (w=4096 sketch is sparse, no dense-broadcast amplification). Resolves the 'raw is cheapest / scale too small' thread with a real workload. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Per the directive that reported metrics must be freshly-run, not looked up: delete all prior (June 14-20) result data — datasets_eval/multisketch/results/ *.json (perfamily accuracy, c1-wire, baselines, gt-*, guard, resource, compound), and the eval-8node June snapshots (RESULTS.md, PHASE2_INTEGRATED_ RESULTS.md, FINDINGS.md, accuracy-raw.csv, fig3/6b/7/8/9/10/11*, costmodel, basesweep, figs/). Kept: the fresh this-session CSVs (f2_debs, f2_wholesketch_cluster*, gos_aniso_cluster) and the run-it-yourself README. The consolidated e2e metrics script (next) regenerates everything fresh. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…-component resources) e2e_metrics.sh merges the two existing harnesses into one run against the REAL multinode ASAPQuery-backend, every number produced fresh: - run_demo.sh lib brings up the asap stack (cold node1 / warm node2 / agents node0+node3) and arm_measure captures per-component CPU/mem + per-node bandwidth + query latency (MetricsQL replay vs node2:9091); - run_perfamily.py (now env-parametrized: E2E_BACKEND / E2E_AGENT_METRICS / E2E_REPLAY_ENDPOINT / --external-stack) replays GT-known slices through the SAME warm backend and scores query ACCURACY vs exact ground truth per family (quantile rel-err / topk recall / freq envelope / cardinality rel-err); - aggregate_report.py + an inline accuracy table → one fresh E2E_METRICS.md. run_perfamily gains env overrides so its scorers run against a remote backend without standing up its own single-host stack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ne fresh report run_demo's arm_measure references measure_per_edge_bandwidth.py/measure_stages.py which no longer exist (stale), so e2e_metrics.sh now uses the WORKING snapshot_resources.sh (per-container CPU/mem + per-node NIC, multinode) + metricsql_replay.py (query latency vs node2:9091). The report is an inline aggregator over this run's snapshot CSVs + replay.jsonl + accuracy JSONs, and degrades gracefully (notes when the family-accuracy prep did not run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The per-cell sketch pipeline ships CMS/CountSketch/DDSketch cells to the backend, which reconstructs the full sketch and answers any query — F2 (‖f‖₂²) is just one such query. The dedicated geometric-CDM F2 monitor (safe-zone gating to save bandwidth) is redundant with that design, since the sketch is shipped anyway for the other queries. Remove it entirely. Edge (asap-precompute-go): delete f2engine.go (+tests), f2driver; unwire FunctionalF2/F2Mode/ParseF2Mode/Spec F2 dims, SetF2Engine/DriveF2Monitor, window.f2Visit, engine.OnRef, and the transport DTOs (Report.Sketch, RefBroadcast). Edge processor: drop functional=f2 + f2_mode. Regenerate monitorpb (drop MonitorReport.sketch, RefBroadcast, CoordToEdge.ref). Docs: delete f2-geometric-monitoring.md + retired ADRs/archive; scrub refs. Per-cell sketches never depended on F2Engine, so the excision is clean. All Go modules build; go vet clean; all tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…y types debs_otlp_map.py maps DEBS 2022 trading events → OTLP for topk/freq/distinct/ quantile with exact offline ground truth; debs_backend_accuracy.sh replays through the real all-families stack and scores each query vs GT. run.py: add --anchor-span-s N — spread the wall-clock-anchored replay across N seconds (distinct ns per point) inside one window, so count-type sketches (value==1.0 per event) keep per-key multiplicity instead of deduping every event of a key onto one identical (series,ts,value) sample. e2e_metrics.sh: consolidated multinode e2e (fresh accuracy + per-component resources), swapping in the working snapshot_resources.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…decision sampling) Row admission is moving to decide exactly once, upstream of serialization (in the OTel-Go SDK, before the raw sample is even built) rather than being independently re-derived at two pipeline stages. The entire reason ConsistentSampler/otlpfilter existed — reproducibility across independent evaluation sites — no longer applies, so delete the whole apparatus: - asap-precompute-go/otlpfilter/: the pre-decode wire filter, deleted whole. - sketches/{cms,countsketch,ddsketch}.go: drop SetSampleIdentity + consistent* fields; each wrapper reverts to its native GeometricSampler-only path. - precompute.go/window.go: drop the SampleIdentitySetter interface + the per-observation identity-threading hook. - monitor/engine.go: drop SetSampleGrantHook (its only caller was the otlpfilter grant wiring); grantedSampleP storage — the independent path feeding the wrapper's own native sampler via applyGrantedSampleP — is untouched and remains the fallback for traffic without SDK-side sampling. - asapedgeprocessor/warm_sketch.go: drop the OnGrant → otlpfilter.Upsert wiring and the now-uncalled wireSampleRows helper. - SDK aggregators (countsketch.go, countminsketch.go): swap ConsistentSampler for GeometricSampler — same per-series, window-salted seed, same RowSampler interface, so the call sites (measure()) are unchanged. All Go modules build; go vet clean; all tests pass (asap-precompute-go, grpcclient, asapedgeprocessor, opentelemetry-go/sdk/metric). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…t fnv64(metric) CDM and the sketch-DB are one system, not two independent identity spaces. The edge's AggID was FNV-1a-64 of the metric name only; the backend's policy identity (PolicyFingerprint, crates/asap_types/src/ policy_fingerprint.rs) is xxh64(seed=0) over the full AggregationConfig (metric, aggregation_type, sub_type, parameters, grouping/aggregated/ rollup labels, window cadence, spatial_filter) — a completely different hash over different inputs, so the two never agreed for the same policy. Add PolicyFingerprint (asap-precompute-go/policyfingerprint.go): a byte-for-byte Go port of the Rust hash, verified against real Rust ground truth (two fixtures — CountSketch with item_label/heap/multi-label grouping, and plain CountMinSketch — both match exactly). Wire it into warm_sketch.go's AggID computation via new backendAggregationType/ backendAggregationParameters helpers mirroring the control plane's sketch_kind_to_backend_type/sketch_params_to_json byte-for-byte (w=cols, d=rows, with_heap only on CountSketch's params, matching the Rust source). Add SpatialFilter to MetricFamily (config.go) — AggregationConfig. spatial_filter has no edge-local representation today; without it, any policy with a non-empty spatial_filter would silently desync this edge's AggID from the backend's. Empty for every family configured today. Add AggregationIdentity (aggregationidentity.go): (AggID, Filter) — AggID is the PolicyFingerprint (per-policy, content-addressed); Filter is the concrete AggregateBy label values (per-group, mirrors groupKeyBytes). Together they identify one physical collector-side sketch instance (the same granularity as the backend's sid) without needing the backend's allocated sid integer — sid is minted by SeriesIdResolver (a counter, requires a round-trip), whereas (AggID, Filter) is fully local. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

AggregationRouter (asap-precompute-go/aggregationrouter.go): maps a series' attributes to the target collector-side AggregationIdentity it folds into plus that aggregation's row fan-out. AggregationPolicy declares the SDK's local knowledge of one collector-side policy (the same shape as ASAPQuery-backend's AggregationConfig) and derives its AggID (PolicyFinger- print) once; .Router() returns a router that projects each series' attrs onto GroupingLabels to get AggregationIdentity.Filter — no per-call hash recomputation, no backend round-trip. liveSampleGrant (opentelemetry-go-patch/sdk/metric/internal/aggregate/ livesamplegrant.go): maintains a live, coordinator-granted sample_p for one AggregationIdentity by dialing the coordinator DIRECTLY — the same monitor.Engine + grpcclient.Client wiring otel-app/sample_controller.go already proves out, just embedded in the SDK process instead of a separate app binary. reportOccurrence()/currentP() mirror sample_controller's observe()/currentP() split precisely: report the rate signal on every occurrence (before the row-admission decision), and roll the window + re-read the grant lazily when the wall clock crosses a boundary. Lives in the SDK module, not asap-precompute-go's core module — it imports monitor/grpcclient, a deliberately separate nested Go module so the gRPC dependency tree never contaminates the core runtime's module graph (mirrors how otel-app depends on both asap-precompute-go and .../monitor/grpcclient as two separate requires). AggregationRouter/PolicyFingerprint/ AggregationIdentity have no such dependency and correctly stay in the core module for both edge and SDK to share. All three touched modules (asap-precompute-go, opentelemetry-go-patch/sdk/ metric via the restored submodule, asapedgeprocessor) build, vet, and test clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e grant into a real Aggregation type The type an app selects via View: decides row admission PER RAW OCCURRENCE at Add()/Record() time, before the occurrence would ever be serialized. An occurrence admitting no row is discarded — never buffered, never exported. An occurrence admitting >=1 row is exported INDIVIDUALLY, never pre-merged with any other occurrence (an earlier design that summed admitted values into one per-row vector per collect interval was wrong: different occurrences carry different keys that map to different columns at the collector, and merging their values destroys the key needed to route each contribution to its own column — verified by a new test, TestRowSampledSketch_DifferentKeysNeverMerged). - metricdata/data.go: RowSampledSketch[N]/RowSampledSketchDataPoint[N] — the one Aggregation in this package whose point count per collect is NOT one-per-series; it's however many raw occurrences were admitted (zero to many). AdmittedRows is a per-occurrence bitmask; Value is raw/unscaled (consumer rescales x1/SampleP); column selection is deliberately absent — that's the collector's job, unrelated to sampling. - aggregation.go: AggregationRowSampledSketch{Router, CoordinatorURL, EdgeID, WindowSizeSecs, BootstrapSampleP}. - internal/aggregate/rowsampledsketch.go: rowSampledSketchValues groups state (GeometricSampler + liveSampleGrant + pending buffer) by precompute.AggregationIdentity — NOT by series, NOT by metric — matching NitroSketch's actual model (sampling a stream feeding ONE sketch). The sampler only Resets when the live-granted p actually changes (a window roll), not every call. cumulative() aliases delta(): there is no sensible "value since a fixed start" for a growing list of individually admitted raw occurrences. - aggregate.go / pipeline.go: Builder[N].RowSampledSketch + the AggregationRowSampledSketch dispatch/compatibility wiring, matching every other sketch family's pattern exactly. 5 new tests cover: p=1.0 admits everything, p<=0 drops everything, distinct AggregationIdentity values get independent samplers, an unrouted series is a pure no-op, and the core correctness property above. Full go.opentelemetry.io/otel/sdk/metric suite (build + vet + test) is green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…for SDK-decided row admission CMSWrapper/CountSketchWrapper.ApplyAdmittedOccurrence(key, value, admittedRows, sampleP) is the collector-side counterpart to the SDK's AggregationRowSampledSketch: given a row-admission bitmask the SDK already decided (upstream, before the occurrence was ever serialized), apply it directly via sketchlib-go's new UpdateStringAtRows / InsertWithHashAtRows — never re-derives admission, never touches w.sampleP or the envelope (the 1/sampleP correction is baked into the cell at insert, matching the existing *SampledPerRow methods' "exact envelope, no double-correct" contract — a query-time consumer must not also rescale by envelope p). Depends on sketchlib-go's UpdateStringAtRows/InsertWithHashAtRows (local replace; no go.mod/go.sum change needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…porters Add the metricdata.RowSampledSketch dispatch case and its transform functions (reserved-attribute encoding of AdmittedRows/Rows/SampleP onto the standard NumberDataPoint/Gauge shape) to both otlpmetricgrpc and otlpmetrichttp, wiring the SDK-side aggregator to real wire export. Also backfills internal/shared/otlp/otlpmetric/transform/metricdata.go.tmpl, which had drifted out of sync with every prior custom sketch family (KLLSketch/CountSketch/CountMinSketch/HLLSketch) added directly to the generated files. The template is now byte-identical to the more complete otlpmetricgrpc generated file (source of truth for future hand-sync), since no gotmpl invocation exists anywhere in this project's actual build.

…only) otlpmetricgrpc's transform layer replaces a data point's attributes with a bare SeriesId once one is assigned (seriesIdentity helper); otlpmetrichttp never got this — it always sent the full attribute set with SeriesId left at 0, a wire-behavior divergence between the two exporters that predates this change. Regenerating otlpmetrichttp from the now-unified template (see prior commit) fixes it in the same motion as keeping both exporters template-derived. Restores 4 "See metrics.proto's reserved N line" doc comments that existed only in the old otlpmetrichttp file and would otherwise have been lost when unifying onto one template.

…rocessor Completes the SDK->collector wire path for AggregationRowSampledSketch: ConsumeMetrics now recognizes the 3 reserved attributes the SDK's OTLP transform stamps on a row-sampled raw occurrence (__asap_row_sampled_admitted_rows/_rows/_sample_p), strips them before they can leak into series identity (gorilla key / AggregateBy grouping / emitted labels), and routes the sample through sketchAggregator.observe's new rowSampled path instead of the normal aggregate-and-insert path. Row-sampled points are never cold-archived (they're pre-sampled fragments of a raw occurrence, not real aggregate samples). Plumbing: precompute.ObservationValue grows RowSampled/AdmittedRows/ SampleP fields (zero-value default, fully backward compatible); CMSObserver and CountSketchObserver branch on RowSampled to call Sketch.ApplyAdmittedOccurrence (applying the SDK's admission bitmask verbatim) instead of InsertHash/UpdateString. Families with no *AtRows sketchlib primitive (DDSketch/KLL/HLL — not row-replicated matrices) drop a row-sampled observation rather than silently misapplying an unrelated observer, since that can only happen on an SDK/collector AggID disagreement.

zzylol · 2026-07-16T16:40:31Z

Split into a reviewable 4-PR stack (verified byte-for-byte identical reconstruction of this branch's tree): #516 (cleanup) → #517 (GOS per-cell delta) → #518 (NitroSketch SDK-side row-admission sampling) → #519 (eval + benchmarks + docs). Closing this monolithic PR in favor of the stack.

Moves row-admission sampling for CMS/CountSketch from the collector into the OTel-Go SDK: the SDK decides, per RAW OCCURRENCE at Record() time, which rows of the TARGET PHYSICAL SKETCH get updated (matching NitroSketch's one-update-stream-per-sketch model), ships the occurrence + admission bitmask over OTLP, and the collector applies it verbatim instead of re-deriving its own sampling decision. - asap-precompute-go: PolicyFingerprint (content-addressed, byte-for-byte match with ASAPQuery-backend's Rust implementation), AggregationIdentity (AggID + Filter — one physical sketch instance per AggregateBy group), AggregationRouter. - Fixes a real bug: this edge's AggID was fnv64(metric name) — completely different from the backend's PolicyFingerprint, silently desyncing CDM identity from the sketch-DB's materialized-view identity for the same policy. Now computed identically on both sides. - sketchlib-go primitives (UpdateStringAtRows/InsertWithHashAtRows) wired through ApplyAdmittedOccurrence on CMSWrapper/CountSketchWrapper, and a per-row (not whole-item) geometric admission swap on CountSketch's plain UpdateString path. - SDK: AggregationRowSampledSketch aggregation type (liveSampleGrant mirrors otel-app/sample_controller.go, dialing the coordinator directly), new metricdata.RowSampledSketch[N] wire type. - OTLP wire transport: both otlpmetricgrpc/otlpmetrichttp dispatch RowSampledSketch to a standard Gauge/NumberDataPoint (no new proto message — a row-sampled point carries no sketch state, just one raw occurrence + 3 scalars stamped as reserved attributes). - Collector: asapedgeprocessor decodes the reserved attributes, strips them before series identity, and routes to ApplyAdmittedOccurrence; drops (rather than misapplies) a row-sampled observation against a family with no *AtRows primitive (DDSketch/KLL/HLL aren't row-replicated matrices) — defensive against an SDK/collector AggID disagreement. - asap-precompute-rs: matching coordinated producer-side sampling for the Rust/OTAP path. Part of splitting #515 into a reviewable stack (cleanup -> GOS -> NitroSketch sampling -> eval/docs). Builds/vets/tests clean standalone: asap-precompute-go, asapedgeprocessor, sdk/metric, otlpmetricgrpc, otlpmetrichttp.

Evaluation for the GOS + NitroSketch sampling work in this stack: - DEBS 2022 real-backend query-accuracy harness (all 4 sketch query types), Google cluster trace integrated ε-sweep (accuracy·latency·freshness·resources, single-node + cluster), C1 bandwidth/encoding-factor experiments (DDSketch/HLL vs raw+gzip), anisotropic-vs-isotropic per-cell delta measurement (historical — anisotropic itself was later removed from #517 pending an Activity_j redesign; this eval data predates that and is kept for the record), raw-vs-sketch cardinality crossover. - otel_collector_benchmark/epsilon_floor: ε-floor vs NitroSketch benchmark, in-tree. - docs/gos-eval-results.md, docs/phase-2.md: recorded eval results and phase-2 implementation history. docs/design-gos-unified-edge-telemetry.md and docs/sampling-cdm-gos-derivations.md moved to #517 (the GOS PR they actually describe) instead of living here. Part of splitting #515 into a reviewable stack (cleanup -> GOS -> NitroSketch sampling -> eval/docs) — the last PR in the stack.

zzylol and others added 30 commits June 18, 2026 07:19

docs: add sampling CDM GOS derivations

052faba

Tighten sampling CDM derivation docs

44c2d3e

Clarify NitroSketch sampling model

d37aea8

Align sampling design with SDK counter updates

f945056

zzylol and others added 21 commits July 6, 2026 11:14

zzylol closed this Jul 16, 2026

zzylol deleted the feat/gos-unified-monitoring branch July 17, 2026 20:08

Navigation Menu

feat: GOS unified edge telemetry — per-cell sketch pipeline, SDK-side NitroSketch row admission, F2 removal - #515

feat: GOS unified edge telemetry — per-cell sketch pipeline, SDK-side NitroSketch row admission, F2 removal#515
zzylol wants to merge 82 commits into
mainfrom
feat/gos-unified-monitoring

zzylol commented Jul 16, 2026

Uh oh!

zzylol commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zzylol commented Jul 16, 2026

Summary

What's actually live (net of two mid-branch pivots)

Key components

Test plan

Uh oh!

zzylol commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant