feat(gos): CountSketch insert-time isotropic delta gate - #524
Merged
Merged
Conversation
Third PR in the split stack (docs #520 -> transport #521 -> this). Converts CountSketch's per-cell delta gating from the periodic decode-prev-diff sub-window model to insert-time detect+reset, per design-gos-unified-edge-telemetry.md §11 — no octosketch package reuse, purpose-built for this mechanism's own semantics. - sketchlib-go CountSketch.UpdateStringGOS (separate commit/PR #69): applies the hashed update then immediately checks each touched cell against threshold; a crossing resets it to 0 in place and is reported, so the delta-since-last-send IS the cell's value right before reset — no separate per-cell accumulator needed. - CountSketchWrapper (countsketch.go): gosEpsilon/gosSites + SetGosMode, a gosDirty list drained by ComputeDeltaAgainst's new GOS branch (drainGosDelta — never decodes `prev`, unlike the fixed-threshold path it leaves untouched), gosWake/ConsumeWakeSignal for the runtime's narrow wake-signal interface, GosDeltaThreshold using the sketch's own incrementally-tracked L2 (QuerySum2 median-of-rows, O(rows)) instead of the O(rows*cols) full-matrix scan L2DivergenceSinceEmit did. Anisotropic mode is NOT reintroduced (see docs/design doc §11 open items — its Activity_j definition breaks under per-cell async reset; tracked as a follow-up, not this PR's job). - gos_threshold.go: F2IsotropicThreshold closed form (T=eps*||C||/(2k*sqrt(dw))), isotropic only. - Runtime plumbing (precompute.go/window.go): Precompute.SetWakeHook + windowState.wakeHook, invoked from recordLocked via a narrow wakeSignaler interface after every observation — the out-of-cycle counterpart to the existing monitorHook. applyGosMode re-stamps GOS config on every already-live series at both flush paths (boundary and sub-window) so a control-plane change takes effect promptly; the CountSketch factory (warm_sketch.go) also primes it at series creation so inserts before a brand-new series' first flush are gated too. subWindowShouldEmit/subWindowMarkEmitted bypass Gate 1's O(dw) divergence check for GOS-mode CountSketch series (the dirty-list drain already decided what's worth sending — an empty drain IS "nothing to send"); Gate 1 itself is NOT deleted here since every other family still depends on it until its own GOS PR lands. - sketchAggregator.subWindowEnabled() decouples from SubWindowInterval when GOS is active (PR #521 flagged this as this PR's job): a GOS-only family with no sub-window ticker configured now still runs EmitSubWindow, driven purely by wakeSubWindow. - config.go/config_validate.go/processor.go: gos_delta_epsilon/gos_sites knobs (validated: countsketch + emit_heap=false only, epsilon in (0,1)), and the SetWakeHook wiring at aggregator construction. Test plan (all new, all passing): - sketchlib-go: disabled-passthrough, deterministic single-row crossing+reset (cell AND L2 verified zeroed), and the key correctness property — telescoping reconstruction (applying every reported crossing onto a fresh target matches a reference sketch fed identical inserts without ever resetting). - asap-precompute-go/sketches: empty-drain-returns-nil, wake-signal fires exactly once per crossing batch and clears on read, wrapper-level telescoping reconstruction via ComputeDeltaAgainst+ApplyDelta. - asapedgeprocessor: TestGOSCountSketchInsertWakesFlush — full pipeline, SubWindowInterval UNSET and WindowDuration=1h (neither can fire on its own), feeding CountSketch observations through ConsumeMetrics alone produces a flushed envelope with NO manual wakeSubWindow() call anywhere in the test. - go build/go vet/go test clean on asapedgeprocessor, asap-precompute-go (all subpackages), and sketchlib-go's CountSketch package. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Jul 16, 2026
zzylol
added a commit
that referenced
this pull request
Jul 17, 2026
…dOccurrence ingest.go: decode the 3 reserved wire attributes an SDK running AggregationRowSampledSketch stamps on an individually-admitted raw occurrence (admitted-rows bitmask, row count, sample_p), strip them before building series identity, and skip cold-archiving a row-sampled point (it's a raw occurrence, not an aggregate sample). warm_sketch.go: observe() takes rowSampled/admittedRows/sampleP and, when set, tags the ObservationValue so CMS/CountSketch route through Sketch.ApplyAdmittedOccurrence (the SDK's admission decision applied verbatim) instead of the plain insert path. Families with no *AtRows sketchlib primitive (DDSketch/KLL/HLL) drop rather than silently misapply an unrelated observer — this can only happen if the SDK's AggregationRouter and this collector's AggID disagreed about which family a PolicyFingerprint targets. Built fresh against current main's warm_sketch.go/ingest.go (not merged wholesale from feat/gos-unified-monitoring, which independently regressed the insert-time GOS wake/anisotropic-threshold work already merged here via #524-#532) — only the row-sampled-specific decode/ route logic was ported; the isotropic GOS priming, wakeCh wiring, and sub-window trigger logic already on main are unchanged. Extracted from #518 (stale base); depends on PR-A's aggID/config plumbing (split/pr-aggid-policyfingerprint) — based on that branch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Third PR in the split stack (docs #520 → transport #521 → this).
Converts CountSketch's per-cell GOS delta gating from a periodic
decode-prev-diff sub-window model to insert-time detect + reset, per
docs/design-gos-unified-edge-telemetry.md§11. Purpose-built for thismechanism (no reuse of sketchlib-go's existing OctoSketch worker framework —
its
AdaptiveTau/always-emit semantics differ from GOS's closed-formε-accuracy thresholds).
Depends on sketchlib-go PR #69 (
feat/gos-per-cell-delta), which this PR'scommit
f5a4735(UpdateStringGOS) was added to.What changed
CountSketch.UpdateStringGOS: applies the normal hashedupdate, then immediately checks each touched cell against a threshold — a
crossing resets the cell to 0 in place (L2 adjusted) and is reported. The
cell's value right before reset already equals its delta-since-last-send,
so no separate per-cell accumulator is needed.
gosEpsilon/gosSites+SetGosMode, agosDirtylist drained byComputeDeltaAgainst's new GOS branch(
drainGosDelta— never decodesprev),gosWake/ConsumeWakeSignalforthe runtime's wake-signal interface, and
GosDeltaThresholdnow reads thesketch's own incrementally-tracked L2 (
QuerySum2, O(rows)) instead of anO(rows·cols) full-matrix scan. Anisotropic mode is not reintroduced —
its
Activity_jdefinition breaks under per-cell async reset (open itemin the design doc), a follow-up, not this PR.
Precompute.SetWakeHook+windowState.wakeHook,invoked from
recordLockedafter every observation via a narrowwakeSignalerinterface — the out-of-cycle counterpart to the existingmonitorHook.subWindowShouldEmit/subWindowMarkEmittedbypass Gate 1'sdivergence check for GOS-mode CountSketch (the dirty-list drain already
decided what's dirty); Gate 1 itself stays for every other family until
its own GOS PR lands.
sketchAggregator.subWindowEnabled()now decouples fromSubWindowIntervalwhen GOS is active (flagged as this PR's job inedge(asapedgeprocessor): wake-on-demand sub-window flush (generic plumbing) #521's description) — a GOS-only family with no sub-window ticker
configured still runs
EmitSubWindow, driven purely by the wake.gos_delta_epsilon/gos_sitesconfig knobs, validated (countsketch +emit_heap=falseonly, epsilon in (0,1)).Test plan
AND L2 verified zeroed), telescoping reconstruction (every reported
crossing applied onto a fresh target matches a reference sketch fed
identical inserts without ever resetting).
crossing batch and clears on read, wrapper-level telescoping
reconstruction via
ComputeDeltaAgainst+ApplyDelta.TestGOSCountSketchInsertWakesFlush— fullpipeline,
SubWindowIntervalunset andWindowDuration=1h(neither can fire on its own); feeding CountSketch observations
through
ConsumeMetricsalone produces a flush, with no manualwakeSubWindow()call anywhere in the test.go build/go vet/go testclean onasapedgeprocessor,asap-precompute-go(all subpackages), and sketchlib-go'sCountSketchpackage.🤖 Generated with Claude Code