feat(monitor): dynamic coordinator→edge sampling coupling (edge half) - #481
Merged
Merged
Conversation
Wires the edge side of the distributed-NitroSketch dynamic coupling: the edge
reports its observed per-window rate so the coordinator can allocate per-edge
update-sampling probabilities, and applies the coordinator-granted sample_p to
its sampling-capable sketch wrappers at the next epoch boundary.
Proto (monitor.proto + regenerated bindings): SlackGrant.sample_p (field 6,
coord→edge) + MonitorReport.rate (field 8, edge→coord).
- types.go: Report.Rate (observed items/window).
- grpcclient/client.go: map outbound Report.Rate → proto rate; inbound proto
SlackGrant.sample_p → monitor.Grant.SampleP.
- engine.go: monitorState tracks obsCount (per-epoch rate) + grantedSampleP;
Observe increments obsCount; sendReportLocked stamps Rate; OnGrant stores
grant.SampleP; grantedSampleP survives EpochReset (allocation persists until
re-granted) but is cleared by ForceReregister (restarted coordinator). New
GrantedSampleP(aggID) accessor (defaults 1.0).
- Apply at epoch boundary (never mid-window): tumbling rotation drains the
series map and next-window sketches are created lazily on first observe, so
the granted p is stamped onto each brand-new empty wrapper at birth via a new
monitorSampleHook (window.go admitSeriesLocked). The in-flight window keeps
its existing p; one p governs the whole next window and every merge operand.
- Family gate (two layers, no panic, no import cycle): precompute
applyGrantedSampleP installs the hook ONLY for CMS/CountSketch/DDSketch by
SketchType; the hook asserts a new SampleSetter{SetSampleP(float64)} interface
(added to CMS/CountSketch/DDSketch wrappers, delegating to WithSampleP) and is
a true no-op for Sum/KLL/HLL (HLL deliberately does NOT implement it — its
max-based estimator gains no CPU from update-sampling and would bias).
Tests: OnGrant stores/clears sample_p; report carries rate; sample_p survives
EpochReset; full apply path (Count-Sketch new-window wrapper → granted p while
in-flight wrapper stays 1.0; Sum no-op); cross-language wire fixtures
(Go-encoded SlackGrant.sample_p / MonitorReport.rate, and a Rust-encoded grant
decoded here) — the gate the Rust side (coupling_wire_compat.rs) mirrors.
Pairs with the coordinator half in ASAPQuery-backend. See
docs/distributed-nitrosketch-coordinated-sampling.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Edge half of the dynamic coordinator↔sampling coupling
Completes the distributed-NitroSketch loop that #480 teed up (the
Grant.SamplePhook +AllocateSampleRates+ the design doc): the edge now reports its observed per-window rate so the coordinator can allocate per-edge update-sampling probabilities, and applies the coordinator-grantedsample_pto its sampling-capable wrappers at the next epoch boundary.Proto (both new fields, regenerated bindings)
SlackGrant.sample_p(field 6) — coord→edge.MonitorReport.rate(field 8) — edge→coord.Edge wiring
monitorState.obsCount(per-epoch), stamped intoReport.Rate.OnGrantstoresgrant.SampleP; persists acrossEpochReset(allocation holds until re-granted), cleared byForceReregister.pis stamped onto each brand-new empty wrapper at birth (window.goadmitSeriesLocked→monitorSampleHook), so onepgoverns the whole next window and every merge operand.SketchType, then aSampleSetterinterface assert); Sum/KLL/HLL are a true no-op (HLL deliberately doesn't implement it — max-based, no CPU win + biased).Tests
OnGrantstore/clear, report carries rate,sample_psurvivesEpochReset, full apply path (Count-Sketch new-window wrapper picks uppwhile the in-flight wrapper stays1.0; Sum no-op), and cross-language wire fixtures (Go-encodedsample_p/ratedecode in Rust; a Rust-encoded grant decodes here). Build +./monitor/ . ./sketches/+ nestedgrpcclientmodule green.Pairs with the coordinator half in ASAPQuery-backend (computes
p_iviaallocate_sample_rates+ the ε-floor and populatesSlackGrant.sample_p). Coupling validated end-to-end across the wire, both directions.🤖 Generated with Claude Code