feat(edge): threshold-driven sub-window delta-sketch transmission - #458
Merged
Merged
Conversation
Rewrites #458's static fixed-interval sub-window producer into a dynamic, divergence-gated one: a series emits a sub-window delta only when its sketch has diverged from the backend's last-acked copy by >= ε in the family's norm, so the backend can answer the OPEN (incomplete) window's query to relative ε at minimal bandwidth — the right behavior when tumbling windows are large (a single boundary emit leaves the open window stale; a blind fixed cadence is wasteful or insufficient). This is the CDM functional-tracking (ε-approximation) branch (Cormode-2013), distinct from the alert coordinator's threshold-exceedance path, and the per-series LOCAL case (no coordinator) of the aggregation taxonomy. Mechanism (kept from #458): EmitSubWindow / ComputeSubWindowDelta (no boundary empty-base reset) / subWindowVisit / serializeSubWindowSeries / the flush check ticker. New: a per-family divergence gate (tight for every family): - Sum: |sum_now − sum_acked| ≥ ε·|sum_now| (value, L1) - CMS: ΔN ≥ ε·N (count, f̂≤f+εN) - KLL/DDSketch: ΔN ≥ ε·N (rank staleness ≤ ΔN) - Count-Sketch: ‖f_now − f_acked‖₂ ≥ ε·‖f_now‖₂ (Frobenius cell-diff) - HLL: |card_now − card_acked| ≥ ε·card_now (relative) Scalar families track the divergence reference on seriesEntry; Count-Sketch snapshots its cell matrix in the wrapper. ε=0 ⇒ emit every tick (#458's fixed mode, backward compatible); first emit of a window always ships full state. Config: SubWindowInterval (check cadence) + SubWindowEpsilon on PrecomputeConfig and the asap_edge processor (sub_window_interval / sub_window_epsilon), validated 0<interval<window and 0≤epsilon<1. Composes with the merged backend per- window_start base rotation (#346): sub-window deltas carry the open window's range and accumulate until the boundary advances window_start. Scope: Go runtime + asap_edge processor (the deployable path). Rust mirror deferred. KLL is not delta-capable, so sub-window emission applies to the delta-capable families. Tests: threshold gating, fixed-mode, disabled-without- delta, Count-Sketch L2 divergence — all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
feat/sub-window-delta-producer
branch
from
June 9, 2026 18:28
8edf4d6 to
0edf3ce
Compare
Sum/count (AggKindSum) and KLL ship FULL state from ComputeDeltaAgainst (their delta ignores the base — they are not incremental). The backend accumulates sub-window emits ADDITIVELY within a window (same window_start), so emitting full state every sub-window tick would over-count Sum / re-merge-inflate KLL. Guard EmitSubWindow with subWindowEmissionSafe so sub-window applies only to the incremental-delta families (DDSketch / Count-Min / Count-Sketch / HLL); Sum/KLL still emit once at the boundary as before. Removes the now-unreachable Sum branch from the divergence gate. Test: TestSubWindow_ExcludesFullStateFamilies asserts Sum and KLL return no sub-window envelopes even with delta + sub-window set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sum's ComputeDeltaAgainst was a stub returning full state, which made Sum unsafe
for sub-window emission (additive backend would over-count). Make Sum a proper
per-window-reset (PWR) incremental delta family, mirroring DDSketch/CMS/etc.:
- ComputeDeltaAgainst now returns {Δsum, Δcount} = current − prev (incremental),
not full state; nil when unchanged.
- SumWrapper implements emptyBaseDeltaSketch.DeltaAgainstEmptyBase (16-byte
{0,0}), so the boundary resets the cached base to empty each window and the
next window's emits are deltas from zero — without this, window N+1's first
delta would be (currentₙ₊₁ − fullₙ), a bogus cross-window subtraction.
Because the boundary ComputeDelta and the sub-window ComputeSubWindowDelta share
the outbound base cache, both now produce deltas against the last emit, so a
window's sub-window emits + boundary emit accumulate additively to the exact
window total (no over-count). Sum is removed from the sub-window exclusion;
subWindowEmissionSafe now gates only KLL (a full-state merge that cannot
subtract). Sum's per-series divergence gate (value: |Δsum| ≥ ε·|sum|) is restored.
Sum metrics WITHOUT delta are unchanged (boundary still serializes full via
Snapshot, never ComputeDelta). Backend compatibility: Sum delta envelopes are
applied via the same additive ApplyDelta + window_start-triggered PWR reset the
backend already uses for the other delta families (#346).
Test: TestSubWindow_Sum_IncrementalReconstruction — three sub-window emits + a
boundary emit reconstruct the window total (175) with no over-count, and a new
window's first emit is a delta from zero (40), proving PWR. Full module green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…AggKind encode)
The runtime made Sum incremental, but Sum sub-window emits never reached the wire
in the asap_edge collector — two gaps, both found via e2e (Sum emitted only at the
boundary while DDSketch/Count-Min fired every sub-window tick):
1. config_validate.go deltaCapable() still excluded FamilySum, so effectiveDelta
forced Sum's DeltaTransmission off → subWindowEnabled()=false → the sub-window
ticker skipped the Sum aggregator. Add FamilySum (KLL stays excluded). Fix the
stale "KLL/Sum cannot delta" comments.
2. serializeSubWindowSeries omitted AggKind, so the Sum envelope (SketchType=
Unspecified, routed by AggKind) failed oteladapter.Encode ("unsupported sketch
type Unspecified") and emitSubWindow silently dropped it. Stamp AggKind like
the boundary serializer does.
Verified end-to-end on the rebuilt asap-otel collector: Sum/count, DDSketch, and
Count-Min all emit threshold-driven sub-window deltas at the ~3s check cadence
within the open 30s window (fixed mode: 11 emits each; ε=0.1: gated ticks
suppressed). Regression test TestSum_SubWindowEmit_EncodesAsSumAgg asserts a Sum
sub-window envelope carries AggKind and encodes to a non-empty SumAgg metric.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…b-window KLL can't subtract (full-state mergeable summary), so it can't ship a subtractive delta — which is why it was excluded. But "full vs delta" is really about WHAT DATA a frame covers (cumulative vs the between-emits segment), not whether the sketch can subtract. So give KLL the same per-frame semantics as the others by the SEGMENT model: after each sub-window emit, reset the sketch, so each frame is a KLL over the DISJOINT data since the last emit. The backend's merge_all then merges the segment rows into the window total exactly as it merges [full,delta, delta] for the subtractive families — no inflation, because disjoint segments share no data (across time AND shard, which also dissolves the earlier shard-merge concern). No backend change needed. The emit trigger is unchanged and family-agnostic: ΔN ≥ ε·N from the external sample count (entry.Count), independent of the sketch's subtractability — so KLL uses the same count-divergence gate as DDSketch. Changes: subWindowSegmentMode(KLL) gates the emit-then-Reset path in EmitSubWindow; KLL added to the asap_edge deltaCapable set so the sub-window ticker fires for it. Test TestSubWindow_KLL_SegmentsReconstruct asserts the merged segment count equals the true observation count (disjoint, no inflation). All sub-window families now: Sum/count, DDSketch, Count-Min, Count-Sketch, HLL (subtractive deltas) + KLL (disjoint segments). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Rewrites this PR from the static fixed-interval sub-window producer into a dynamic, threshold-driven one. A series ships an incremental sub-window delta only when its sketch has diverged from the backend's last-acked copy by ≥ ε in the family's norm — so the backend can answer the open (incomplete) window's query to relative ε at minimal bandwidth.
Why (large tumbling windows)
With a large window, a single boundary emit leaves the open window stale for a long time, and a blind fixed cadence is either wasteful (re-emits stable series) or insufficient (can't react to a burst). Threshold-driven emission bounds the open-window query error to ε regardless of window size, and emits nothing while a series is stable.
Which CDM guarantee this is (Cormode-2013)
This is the functional-tracking / ε-approximation branch — the backend's reconstructed open-window sketch stays within ε·‖f‖ of the true current sketch, so it can query the value to relative ε. It is not the threshold-exceedance/countdown path (that's the separate alert coordinator). It is the per-series local case (no coordinator). See
docs/continuous-monitoring-aggregation-taxonomy.md.How
Threshold mode is a superset of the old fixed mode. The mechanism is retained (
EmitSubWindow,ComputeSubWindowDelta— the no-boundary-empty-base within-window diff,subWindowVisit,serializeSubWindowSeries, the flush check ticker). New is a per-series divergence gate, tight for every family:ΔN ≥ ε·N(f̂ ≤ f + εN)ΔN ≥ ε·N(rank staleness ≤ ΔN; DDSketch value α-relative)‖f_now − f_acked‖₂ ≥ ε·‖f_now‖₂(Frobenius cell-diff)Scalar families track the divergence reference on
seriesEntry; Count-Sketch snapshots its cell matrix in the wrapper.sub_window_epsilon = 0⇒ emit every check tick (the old fixed mode, backward compatible); the first emit of a window always ships full state.Config
PrecomputeConfig.SubWindowInterval(check cadence) +SubWindowEpsilon.asap_edge:sub_window_interval+sub_window_epsilon, validated0 < interval < window_durationand0 ≤ epsilon < 1.Composition with the backend
Composes with the merged per-
window_startbase rotation (#346): sub-window deltas carry the open window's range and accumulate until the boundary advanceswindow_start.Tests
asap-precompute-go: threshold gating (stable series emits nothing; crossing ε emits), first-emit-full, fixed-mode (ε=0) emits every tick, disabled-without-delta, Count-Sketch L2 divergence.asap_edge: config validation of both knobs. All green.Accuracy (proof sketch)
Between emits the backend holds
S_acked; the gate guaranteesD(S_now, S_acked) < ε·‖S_now‖at all times, so the backend's open-window answer is within relative ε. Across k edges the per-edge thresholds compose additively to ε·‖f_global‖ (L1; DDSketch α structural). Full per-family proofs indocs/continuous-monitoring-tumbling-cost-analysis.md.Applicable families
Sub-window emission applies to all the incremental-delta families — Sum/count, DDSketch, Count-Min, Count-Sketch, HLL. Sum was made incremental for this: its
ComputeDeltaAgainstnow ships{Δsum,Δcount}and it opts into per-window-reset (DeltaAgainstEmptyBase), so a window's sub-window emits + boundary emit accumulate additively to the exact window total (TestSubWindow_Sum_IncrementalReconstruction). KLL is excluded (subWindowEmissionSafe): itsComputeDeltaAgainstis a full-state merge (it cannot subtract), so re-emitting every sub-window would re-merge-inflate it — KLL emits once at the boundary as before.Scope / follow-ups
asap_edge(the deployable path). Rustasap-precompute-rsmirror deferred.ApplyDelta+window_start-triggered PWR reset the backend already uses for the other delta families (mvp_report: fix DDSketch family metric name (http_latency_ms → http_requests_total_latency_ms) #346).🤖 Generated with Claude Code