feat(monitor): whole-sketch F2 functional + unify sampling to the ε-floor - #381
Merged
Merged
Conversation
…e-sketch) Adds Functional::F2 (whole-sketch L2 = ‖f‖₂², no per-point key) and publishes the monitor's functional in the streaming-config entry so the edge can auto-learn its reporting mode. MonitorSpec gains a pass-through functional field. The coordinator allocation is unchanged: it allocates on the reported scalar value, and p_i ∝ √(value/rate) IS the F2 allocation when the edge reports value=‖f_i‖² (per data_plane::monitor whole-sketch decision rule, PR #380). This is the B1 (scalar-F2) live path; the sketch-shipping + geometric DistributedF2Monitor (f2.rs) stays as the comm-efficient B2 follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…loor
Retire the per-key √(f_i/rate_i) KKT allocation; the single coordinated-sampling
law for a sketch is the whole-sketch ε-floor p_i = 1/(1+ε²·rate_i).
Why: sampling protects the warm SKETCH, whose point/L2 estimate error is bounded
by the stream norm (rate/L2 — NitroSketch), never by a single key's f(x). So the
only accuracy a per-edge p buys is keeping that edge's mass within ε
(ε_s=√((1−p)/(p·rate))≤ε ⇒ the floor). The √(f/rate) allocation only holds when a
key is EXACT-counted OUTSIDE the sketch — and sampling one exact counter is
pointless — so it is not a valid sketch-sampling regime. The monitored functional
(cms_point/f2/sum) still drives the THRESHOLD (known_value→global_estimate/alert);
it no longer drives sampling.
- coordinator::allocate_p → ε-floor only (drop var_budget + allocate_sample_rates).
- sampling_alloc: delete allocate_sample_rates + uniform_sample_rate (+ tests);
keep epsilon_sample_floor. Module doc states the law + applicability: additive
sketches (Count-Min L1, Count-Sketch L2, DDSketch/KLL rank, Sum); NOT HLL
(update-sampling biases cardinality, not 1/p-correctable).
- f2: delete DistributedF2Monitor::{allocate_p, sampling_var_budget} + EdgeF2.rate;
it is now a pure F2 THRESHOLD monitor (CountSketchF2 + geometric kept).
- mod.rs doc: separate the two orthogonal axes (what-to-monitor vs how-to-sample).
28/28 monitor tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two coherent changes to the CDM monitor, on top of the merged #377/#378/#379:
1. Whole-sketch F2 functional (round-trip)
Adds
Functional::F2and publishes the monitor'sfunctionalin the streaming-config entry so the edge auto-learns its reporting mode.MonitorSpecgains a pass-throughfunctionalfield. F2 = whole-sketch L2 (‖f‖₂², no per-point key, identity(agg_id,"")); the edge reports its localF2_i = Σ_x f(x)².2. Unify coordinated sampling to the whole-sketch ε-floor
Retire the per-key
√(f_i/rate_i)KKT allocation. The single coordinated-sampling law for a sketch is the whole-sketch ε-floorWhy: sampling protects the warm SKETCH, whose point/L2 estimate error is bounded by the stream norm (rate/L2 — NitroSketch), never by a single key's
f(x). The only accuracy a per-edgepbuys is keeping that edge's mass within ε (ε_s=√((1−p)/(p·rate))≤ε ⇒ the floor).√(f/rate)only holds when a key is EXACT-counted outside the sketch — pointless to sample — so it is not a valid sketch-sampling regime. The monitored functional (cms_point/f2/sum) still drives the threshold/alert (known_value→global_estimate); it no longer drives sampling.coordinator::allocate_p→ ε-floor only (dropvar_budget+allocate_sample_rates).sampling_alloc: deleteallocate_sample_rates+uniform_sample_rate; keepepsilon_sample_floor. Module doc states the law + applicability: Count-Min, Count-Sketch, DDSketch, KLL, Sum — NOT HLL (update-sampling biases cardinality, not 1/p-correctable).f2: deleteDistributedF2Monitor::{allocate_p, sampling_var_budget}+EdgeF2.rate→ now a pure F2 threshold monitor (CountSketchF2 + geometric kept).mod.rsdoc: the two orthogonal axes (what-to-monitor vs how-to-sample). (Supersedes the closed docs(monitor): when to use whole-sketch (agg_id) vs single-point (agg_id,key) #380.)Validated live (8-node, both monitor types collapse to one law)
28/28 monitor tests pass; data-plane image rebuilt + re-run, auto-learn + hot-reload green.
🤖 Generated with Claude Code