feat(monitor): distributed-NitroSketch coordinated update-sampling (prototype) - #480
Merged
Merged
Conversation
…rototype) Extends NitroSketch's per-sketch update-sampling to a multi-edge setting: the CDM coordinator allocates each edge a DIFFERENT sampling rate p_i to minimize total edge update CPU at a target merged accuracy, instead of one p everywhere. - docs/distributed-nitrosketch-coordinated-sampling.md — the design (variance composition Var_merged = Σ f_i(1−p_i)/p_i, KKT solution p_i ∝ √(f_i/rate_i), per-family applicability, coordinator hooks, joint CPU/bandwidth budget, honest assessment). - monitor/sampling_alloc.go — AllocateSampleRates (the coordinator's allocation step, p_i ∝ √(f_i/rate_i) via bisection on the variance constraint) + UniformSampleRate (the per-edge-independent baseline). Reference impl to port to the Rust coordinator. - monitor/sampling_alloc_test.go — on a skewed fleet (1 hot edge + quiet tail, key spread evenly) the coordinated allocation saves ~62% edge update work at EQUAL merged variance vs uniform p; flat-fleet test confirms ~0 win (honest). - sketches/countsketch.go — CountSketchWrapper.WithSampleP: geometric update-sampling at the wrapper level (admit w.p. p, upweight by 1/p ⇒ unbiased estimate, no backend rescale needed). Test: sampled p=0.25 estimate within ~1% of true while touching counters ~1/4 as often. - monitor.Grant.SampleP — the wire field the coordinator populates and the edge applies via WithSampleP at EpochReset (orthogonal to LocalSlack, which governs emission/bandwidth). Prototype: the allocation algorithm + edge capability + wire field are done and tested; the coordinator computing p_i from per-edge reports and granting over gRPC (mirrors the existing slack path) is the remaining integration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(pN)) coupling, and the empirical in-collector CPU caveat - Combined bound: when update-sampling and the CDM threshold run together at the edge, the value the CDM tests is an unbiased-but-noisy 1/p-rescaled estimate (sampling error ε_s ≈ √((1−p)/(pN))). For ε-approximation/freshness the bound holds additively (ε_sk + ε_s + ε_cdm, shrinking with N); for the slack-countdown alert the no-missed-crossing safety becomes (1−δ)-probabilistic and needs a confidence-margin inflation. Communication only holds if the threshold band absorbs the jitter — the coupling rule ε_cdm ≳ √((1−p)/(pN)) (don't sample so hard the noise exceeds the CDM tolerance; higher per-series N relaxes it). - Empirical caveat: a multi-shape e2e shows the CPU win does NOT materialize in the OTel collector (edge CPU flat ±noise) while egress drops ~9% — the sketch- update loop is a small fraction vs OTLP decode + key extraction + allocation (allocation-bound, per the repo's jemalloc 2.2× finding). Reframe the objective as egress minimization unless a profile shows the counter loop dominates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…alloc) A producer CPU profile found the dominant edge cost is CMSWrapper.ComputeDeltaAgainst re-materializing the full d×w matrix every sub-window/flush emit: it DeserializeCountMinSketchFromProtoBytes(prev) (allocating+zeroing a matrix) + Snapshot() (re-serialize) just to diff. Under the PWR contract prev is always the empty base, so this reconstructs a zero matrix every emit to subtract zero. Fast path: detect the empty base cheaply (bytes.Equal vs a memoized empty-base envelope — exactly the bytes the SnapshotCache hands back each window) and call sketchlib's new ComputeDeltaAgainstEmpty (ProjectASAP/sketchlib-go#67) instead of the deserialize+diff. The size-clamp (Snapshot) and the non-empty (real prev) path are unchanged, so the emitted bytes are byte-identical (parity test asserts the fast-path payload decodes to exactly-equal Count-Min state). WithSampleP invalidates the memo. This removes the per-emit CMS cost the profiler flagged (~32% of edge CPU under sub-window) — i.e. it's what makes the threshold-driven sub-window producer and the coordinated-sampling emit-gating actually cheap. Requires sketchlib-go#67 (go.mod replace resolves it locally). Bench (5x2048, delta-against-empty): 837us->306us, 978KB->157KB, 65->21 allocs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Prototype of distributed NitroSketch: extends NitroSketch's per-sketch update-sampling to a multi-edge setting, where the CDM coordinator allocates each edge a different sampling rate
p_ito minimize total edge update CPU at a target merged accuracy — instead of onepeverywhere.Contents
docs/distributed-nitrosketch-coordinated-sampling.md— the design: variance compositionVar_merged = Σ f_i(1−p_i)/p_i, the KKT solutionp_i ∝ √(f_i/rate_i), per-family applicability, coordinator hooks, the joint CPU/bandwidth budget with the sub-window emitter, and a skeptical assessment.monitor/sampling_alloc.go—AllocateSampleRates(the coordinator's allocation step,p_i ∝ √(f_i/rate_i)via bisection on the variance constraint) +UniformSampleRate(the per-edge-independent baseline). Reference impl to port to the Rust coordinator.sketches/countsketch.go—CountSketchWrapper.WithSampleP: geometric update-sampling at the wrapper (admit w.p.p, upweight by1/p⇒ unbiased estimate, no backend rescale needed).monitor.Grant.SampleP— the wire field the coordinator populates and the edge applies viaWithSamplePatEpochReset(orthogonal toLocalSlack, which governs emission/bandwidth).Result (the coordination win, quantified)
On a skewed fleet (1 hot edge at 100k/win + 4 quiet at 1k/win, key spread evenly) the coordinated allocation saves ~62% edge update-work at EQUAL merged variance vs uniform
p(hot edgep=0.056, quietp=0.56). A flat-fleet test confirms ~0 win (honest — the win scales with the per-edge rate skew). Count-Sketch sampled atp=0.25estimates within ~1% of true while touching counters ~¼ as often.Scope (prototype)
The allocation algorithm + edge sampling capability + wire field are done and tested. The remaining integration (not in this PR): the coordinator computing
p_ifrom per-edge rate reports and granting over gRPC (mirrors the existing slack path), and — if you prefer uniform-rescale over the wrapper's upweight-on-admit — a Count-Sketch backend1/prescale. Pairs with the backend CMS/HLL1/prescale fix (separate PR).🤖 Generated with Claude Code