feat(otel-app): coordinated-sampling edge foundation — per-monitor report, auto-learn, multi-monitor, register-under-key - #504
Merged
Conversation
… sampling The coordinated edge reported windowValue = monitorValuePerObs(0.5) × count, i.e. a value PROPORTIONAL to the rate, so the coordinator's p_i ∝ √(f_i/rate_i) saw √(0.5·rate/rate)=√0.5 (constant) → uniform p, no differentiation. Add -monitor-key: when set (cms_point heavy-hitter), only events whose series_id equals the key add to the reported VALUE (f_i = the monitored key's frequency), while EVERY event counts toward the RATE (rate_i = total updates). This decouples f_i from rate_i so a fixed-frequency key on a higher-rate edge reports a smaller f_i/rate_i and the coordinator samples it harder (smaller p). Empty key keeps the legacy sum-monitor behaviour (value ∝ rate ⇒ uniform p, correct for a sum). observe() now takes the series_id; the replay hot path passes r.seriesID. Pairs with ASAPQuery-backend#377 (coordinator uses known_value as f_i). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nfig Instead of requiring a static -monitor-key flag, the coordinated edge now learns which key it is monitoring from the CONTROL PLANE: -monitor-config-url points at the data-plane streaming-config (the document the controller POSTs its monitors: config to); on startup the edge fetches it, finds the monitors[] entry whose agg_id matches -monitor-agg-id, and uses that entry's `key` as f_i's key. So the cms_point key flows controller → data-plane streaming-config → edge, closing the loop with the controller's monitor emission (mirrors ASAPQuery-backend#377). -monitor-key remains as a manual override (tests / no config URL); empty key + no config URL = sum-monitor (value ∝ rate ⇒ uniform p). agg_id is read as a full uint64 (no float64 precision loss). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…p=max) An edge's stream can feed SEVERAL cms_point monitors at once (different keys / series). Generalize sample_controller from a single (agg_id, key) to a SET of monitors learned from the controller-pushed config: - per-monitor windowValue f_m (count of that monitor's key) + a SHARED rate; - report EACH monitor under its own agg_id every window (one engine multiplexes agg_ids via Observe(aggID,...)/GrantedSampleP(aggID)); - the warm-sample p is a SINGLE admission for the whole warm stream, so apply p = max_m p_m (least aggressive grant) — this keeps EVERY monitor's variance budget satisfied (a smaller p would over-sample and blow some monitor's V_m). -monitor-config-url now learns ALL monitors (not just the agg_id match); a single -monitor-agg-id/-monitor-key is the fallback/override. fetchMonitors replaces fetchMonitorKey. Builds + go vet clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
zzylol
changed the title
zzylol
changed the title
The coordinated-sampling edge imports asap-precompute-go (monitor.Engine + grpcclient). go.mod replaces it with ../asap-precompute-go, but the Dockerfile never staged it, so go mod tidy failed (open /src/asap-precompute-go/go.mod: no such file). COPY the named build-context to /src/asap-precompute-go; from /src/otel-app the ../asap-precompute-go replace resolves there directly (no sed rewrite, unlike sketchlib-go's two-level path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The coordinator identifies a monitor by (agg_id, key) — a cms_point monitor on sid s0 is keyed (agg_id, "s0"). The engine.Observe calls passed nil as the key, so the edge registered and reported under (agg_id, ""); the coordinator matched no configured monitor and rejected every registration with 'register for unconfigured monitor — ignored', leaving the edge ungranted (p stuck at 1). Thread each monitor's key (cached as keyBytes to avoid a per-observe allocation on the replay hot path) into both Observe calls so registration/reports carry the configured key. Empty key (sum monitor) is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mergeFlagOverrides copied coordinator-url/monitor-agg-id/edge-id from the parsed flags into the effective config but silently dropped -monitor-config-url and -monitor-key. So both stayed empty regardless of the CLI: the auto-learn block (guarded by MonitorConfigURL != "") was skipped and the fallback monitor took key="". The edge then registered under (agg_id, "") while the coordinator's cms_point monitor is keyed (agg_id, "s0") — every registration was rejected as an unconfigured monitor and the edge never got a grant. Merge both fields. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
zzylol
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The edge half of coordinated sampling (pairs with ASAPQuery-backend#377, the coordinator half).
Terminology: each cms_point monitor tracks the frequency of one series id (sid) — a point in the Count-Min sketch, internally the CMS
key— and is identified by an aggregation_id (agg_id). The quantity that drives the allocation is the per-edge frequency of that sid,f_i, reported under its agg_id — not the edge's total update rate. Three commits:1. Report per-sid frequency
f_i(not0.5×count)The edge reported
windowValue = 0.5×count(∝ rate), so the coordinator saw√(f_i/rate_i)=√0.5(constant) → uniformp. New-monitor-key(the sid): only events matching the monitored sid add to the reported value (f_i= that sid's frequency), while every event counts toward the rate — decoupling them sop_i ∝ √(f_i/rate_i)differentiates (a fixed-frequency sid on a higher-rate edge → smallerp). Empty sid = legacy sum monitor (value ∝ rate, correct for a sum).2. Learn the sid from the controller-pushed config (close the loop)
Instead of a static flag, the edge learns which sid it monitors from the control plane:
-monitor-config-urlpoints at the data-plane streaming-config (where the controller POSTs itsmonitors:, each{agg_id, key}); on startup the edge fetches it and reads the sid + agg_id.-monitor-keybecomes a manual override. (agg_idparsed as a fulluint64.)3. Multi-monitor (several sids / several agg-ids on one stream)
An edge's stream can feed several cms_point monitors at once.
sample_controllergeneralizes from a single(agg_id, sid)to a SET:f_m(count of that monitor's sid) + a shared rate;agg_id(one engine multiplexes agg_ids);pis a single admission for the whole warm stream, so applyp = max_m p_m(least aggressive grant) — which keeps every monitor's variance budget satisfied (a smallerpwould over-sample and blow someV_m).Builds +
go vetclean.🤖 Generated with Claude Code
Live validation (Fig 9, 8-node CloudLab) — two more correctness fixes
End-to-end testing on hardware surfaced two bugs that kept the coordinator from ever matching the edge, each fixed here:
nil. The coordinator identifies a monitor by(agg_id, key); a cms_point monitor ons0is keyed(agg_id,"s0"). Theengine.Observecalls passednil, so the edge registered as(agg_id,"")→register for unconfigured monitor — ignored→ no grant. Now threads the monitor key (cached askeyBytesto avoid a per-observe allocation on the replay hot path).-monitor-config-url/-monitor-keyflag overrides.mergeFlagOverridescopiedcoordinator-url/monitor-agg-id/edge-idbut silently dropped these two, so the effective key was always""(auto-learn skipped, fallback empty). Now merged.With these, the edge auto-learns
key=s0from the controller config and the coordinator allocates differentiated grantsp = 0.0010 / 0.0022 / 0.0049(hot/med/quiet, rate 400:80:16) — ratio 1:2.2:4.9 vs the √rate law 1:2.24:5. See ASAPCollector#503 (Fig 9).