Skip to content

feat(edge): threshold-driven sub-window delta-sketch transmission - #458

Merged
zzylol merged 5 commits into
mainfrom
feat/sub-window-delta-producer
Jun 11, 2026
Merged

zzylol merged 5 commits into
mainfrom
feat/sub-window-delta-producer

Conversation

@zzylol

@zzylol zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor

What

Rewrites this PR from the static fixed-interval sub-window producer into a dynamic, threshold-driven one. A series ships an incremental sub-window delta only when its sketch has diverged from the backend's last-acked copy by ≥ ε in the family's norm — so the backend can answer the open (incomplete) window's query to relative ε at minimal bandwidth.

Why (large tumbling windows)

With a large window, a single boundary emit leaves the open window stale for a long time, and a blind fixed cadence is either wasteful (re-emits stable series) or insufficient (can't react to a burst). Threshold-driven emission bounds the open-window query error to ε regardless of window size, and emits nothing while a series is stable.

Which CDM guarantee this is (Cormode-2013)

This is the functional-tracking / ε-approximation branch — the backend's reconstructed open-window sketch stays within ε·‖f‖ of the true current sketch, so it can query the value to relative ε. It is not the threshold-exceedance/countdown path (that's the separate alert coordinator). It is the per-series local case (no coordinator). See docs/continuous-monitoring-aggregation-taxonomy.md.

How

Threshold mode is a superset of the old fixed mode. The mechanism is retained (EmitSubWindow, ComputeSubWindowDelta — the no-boundary-empty-base within-window diff, subWindowVisit, serializeSubWindowSeries, the flush check ticker). New is a per-series divergence gate, tight for every family:

family norm gate
Sum L1 (value) `
Count-Min L1 (count) ΔN ≥ ε·N (f̂ ≤ f + εN)
KLL / DDSketch rank ΔN ≥ ε·N (rank staleness ≤ ΔN; DDSketch value α-relative)
Count-Sketch L2 ‖f_now − f_acked‖₂ ≥ ε·‖f_now‖₂ (Frobenius cell-diff)
HLL relative `

Scalar families track the divergence reference on seriesEntry; Count-Sketch snapshots its cell matrix in the wrapper. sub_window_epsilon = 0 ⇒ emit every check tick (the old fixed mode, backward compatible); the first emit of a window always ships full state.

Config

  • PrecomputeConfig.SubWindowInterval (check cadence) + SubWindowEpsilon.
  • asap_edge: sub_window_interval + sub_window_epsilon, validated 0 < interval < window_duration and 0 ≤ epsilon < 1.

Composition with the backend

Composes with the merged per-window_start base rotation (#346): sub-window deltas carry the open window's range and accumulate until the boundary advances window_start.

Tests

asap-precompute-go: threshold gating (stable series emits nothing; crossing ε emits), first-emit-full, fixed-mode (ε=0) emits every tick, disabled-without-delta, Count-Sketch L2 divergence. asap_edge: config validation of both knobs. All green.

Accuracy (proof sketch)

Between emits the backend holds S_acked; the gate guarantees D(S_now, S_acked) < ε·‖S_now‖ at all times, so the backend's open-window answer is within relative ε. Across k edges the per-edge thresholds compose additively to ε·‖f_global‖ (L1; DDSketch α structural). Full per-family proofs in docs/continuous-monitoring-tumbling-cost-analysis.md.

Applicable families

Sub-window emission applies to all the incremental-delta families — Sum/count, DDSketch, Count-Min, Count-Sketch, HLL. Sum was made incremental for this: its ComputeDeltaAgainst now ships {Δsum,Δcount} and it opts into per-window-reset (DeltaAgainstEmptyBase), so a window's sub-window emits + boundary emit accumulate additively to the exact window total (TestSubWindow_Sum_IncrementalReconstruction). KLL is excluded (subWindowEmissionSafe): its ComputeDeltaAgainst is a full-state merge (it cannot subtract), so re-emitting every sub-window would re-merge-inflate it — KLL emits once at the boundary as before.

Scope / follow-ups

🤖 Generated with Claude Code

Rewrites #458's static fixed-interval sub-window producer into a dynamic,
divergence-gated one: a series emits a sub-window delta only when its sketch has
diverged from the backend's last-acked copy by >= ε in the family's norm, so the
backend can answer the OPEN (incomplete) window's query to relative ε at minimal
bandwidth — the right behavior when tumbling windows are large (a single
boundary emit leaves the open window stale; a blind fixed cadence is wasteful or
insufficient). This is the CDM functional-tracking (ε-approximation) branch
(Cormode-2013), distinct from the alert coordinator's threshold-exceedance path,
and the per-series LOCAL case (no coordinator) of the aggregation taxonomy.

Mechanism (kept from #458): EmitSubWindow / ComputeSubWindowDelta (no boundary
empty-base reset) / subWindowVisit / serializeSubWindowSeries / the flush check
ticker. New: a per-family divergence gate (tight for every family):
  - Sum:        |sum_now − sum_acked| ≥ ε·|sum_now|   (value, L1)
  - CMS:        ΔN ≥ ε·N                                (count, f̂≤f+εN)
  - KLL/DDSketch: ΔN ≥ ε·N                              (rank staleness ≤ ΔN)
  - Count-Sketch: ‖f_now − f_acked‖₂ ≥ ε·‖f_now‖₂       (Frobenius cell-diff)
  - HLL:        |card_now − card_acked| ≥ ε·card_now    (relative)
Scalar families track the divergence reference on seriesEntry; Count-Sketch
snapshots its cell matrix in the wrapper. ε=0 ⇒ emit every tick (#458's fixed
mode, backward compatible); first emit of a window always ships full state.

Config: SubWindowInterval (check cadence) + SubWindowEpsilon on PrecomputeConfig
and the asap_edge processor (sub_window_interval / sub_window_epsilon), validated
0<interval<window and 0≤epsilon<1. Composes with the merged backend per-
window_start base rotation (#346): sub-window deltas carry the open window's
range and accumulate until the boundary advances window_start.

Scope: Go runtime + asap_edge processor (the deployable path). Rust mirror
deferred. KLL is not delta-capable, so sub-window emission applies to the
delta-capable families. Tests: threshold gating, fixed-mode, disabled-without-
delta, Count-Sketch L2 divergence — all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the feat/sub-window-delta-producer branch from 8edf4d6 to 0edf3ce Compare June 9, 2026 18:28
@zzylol zzylol changed the title feat(edge): static sub-window delta producer (paper §4.2 "fixed" mode) feat(edge): threshold-driven sub-window delta-sketch transmission Jun 9, 2026
zzylol and others added 4 commits June 9, 2026 12:46
Sum/count (AggKindSum) and KLL ship FULL state from ComputeDeltaAgainst (their
delta ignores the base — they are not incremental). The backend accumulates
sub-window emits ADDITIVELY within a window (same window_start), so emitting full
state every sub-window tick would over-count Sum / re-merge-inflate KLL. Guard
EmitSubWindow with subWindowEmissionSafe so sub-window applies only to the
incremental-delta families (DDSketch / Count-Min / Count-Sketch / HLL); Sum/KLL
still emit once at the boundary as before. Removes the now-unreachable Sum branch
from the divergence gate. Test: TestSubWindow_ExcludesFullStateFamilies asserts
Sum and KLL return no sub-window envelopes even with delta + sub-window set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sum's ComputeDeltaAgainst was a stub returning full state, which made Sum unsafe
for sub-window emission (additive backend would over-count). Make Sum a proper
per-window-reset (PWR) incremental delta family, mirroring DDSketch/CMS/etc.:

  - ComputeDeltaAgainst now returns {Δsum, Δcount} = current − prev (incremental),
    not full state; nil when unchanged.
  - SumWrapper implements emptyBaseDeltaSketch.DeltaAgainstEmptyBase (16-byte
    {0,0}), so the boundary resets the cached base to empty each window and the
    next window's emits are deltas from zero — without this, window N+1's first
    delta would be (currentₙ₊₁ − fullₙ), a bogus cross-window subtraction.

Because the boundary ComputeDelta and the sub-window ComputeSubWindowDelta share
the outbound base cache, both now produce deltas against the last emit, so a
window's sub-window emits + boundary emit accumulate additively to the exact
window total (no over-count). Sum is removed from the sub-window exclusion;
subWindowEmissionSafe now gates only KLL (a full-state merge that cannot
subtract). Sum's per-series divergence gate (value: |Δsum| ≥ ε·|sum|) is restored.

Sum metrics WITHOUT delta are unchanged (boundary still serializes full via
Snapshot, never ComputeDelta). Backend compatibility: Sum delta envelopes are
applied via the same additive ApplyDelta + window_start-triggered PWR reset the
backend already uses for the other delta families (#346).

Test: TestSubWindow_Sum_IncrementalReconstruction — three sub-window emits + a
boundary emit reconstruct the window total (175) with no over-count, and a new
window's first emit is a delta from zero (40), proving PWR. Full module green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…AggKind encode)

The runtime made Sum incremental, but Sum sub-window emits never reached the wire
in the asap_edge collector — two gaps, both found via e2e (Sum emitted only at the
boundary while DDSketch/Count-Min fired every sub-window tick):

  1. config_validate.go deltaCapable() still excluded FamilySum, so effectiveDelta
     forced Sum's DeltaTransmission off → subWindowEnabled()=false → the sub-window
     ticker skipped the Sum aggregator. Add FamilySum (KLL stays excluded). Fix the
     stale "KLL/Sum cannot delta" comments.
  2. serializeSubWindowSeries omitted AggKind, so the Sum envelope (SketchType=
     Unspecified, routed by AggKind) failed oteladapter.Encode ("unsupported sketch
     type Unspecified") and emitSubWindow silently dropped it. Stamp AggKind like
     the boundary serializer does.

Verified end-to-end on the rebuilt asap-otel collector: Sum/count, DDSketch, and
Count-Min all emit threshold-driven sub-window deltas at the ~3s check cadence
within the open 30s window (fixed mode: 11 emits each; ε=0.1: gated ticks
suppressed). Regression test TestSum_SubWindowEmit_EncodesAsSumAgg asserts a Sum
sub-window envelope carries AggKind and encodes to a non-empty SumAgg metric.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…b-window

KLL can't subtract (full-state mergeable summary), so it can't ship a subtractive
delta — which is why it was excluded. But "full vs delta" is really about WHAT
DATA a frame covers (cumulative vs the between-emits segment), not whether the
sketch can subtract. So give KLL the same per-frame semantics as the others by
the SEGMENT model: after each sub-window emit, reset the sketch, so each frame is
a KLL over the DISJOINT data since the last emit. The backend's merge_all then
merges the segment rows into the window total exactly as it merges [full,delta,
delta] for the subtractive families — no inflation, because disjoint segments
share no data (across time AND shard, which also dissolves the earlier
shard-merge concern). No backend change needed.

The emit trigger is unchanged and family-agnostic: ΔN ≥ ε·N from the external
sample count (entry.Count), independent of the sketch's subtractability — so KLL
uses the same count-divergence gate as DDSketch.

Changes: subWindowSegmentMode(KLL) gates the emit-then-Reset path in
EmitSubWindow; KLL added to the asap_edge deltaCapable set so the sub-window
ticker fires for it. Test TestSubWindow_KLL_SegmentsReconstruct asserts the
merged segment count equals the true observation count (disjoint, no inflation).
All sub-window families now: Sum/count, DDSketch, Count-Min, Count-Sketch, HLL
(subtractive deltas) + KLL (disjoint segments).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 4ad2b9f into main Jun 11, 2026
@zzylol
zzylol deleted the feat/sub-window-delta-producer branch June 11, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant