feat(gos): DDSketch sparse wire fix + insert-time delta gate (T=εN/(kB)) - #528
Merged
Merged
Conversation
Converts DDSketch to the GOS insert-time delta mechanism, following the CountSketch precedent (commit 9a0b401). Instead of the periodic decode-prev-snapshot-and-diff sub-window scan, each bucket touch now checks its own accumulated-since-last-send count against a closed-form threshold; on crossing, the value is captured as the delta, the bucket is reset to 0 in place, and the flush loop is woken immediately. DDSketch's CDM isotropic threshold (derivations §8.4) is the L1 value-range-count form T=ε·N/(k·B), where N is the sketch's total count and B is the number of POPULATED buckets. B is read from sketchlib-go DDSketch.PopulatedBuckets() — tracked incrementally and exactly (first-touch increment / reset decrement), never assumed: the derivation is explicit that understating B silently VIOLATES the staleness guarantee (scales as ε·N·(B_actual/B_assumed), only bounded when B_assumed >= B_actual), so an assumed constant is not safe. Changes: - sketches/gos_threshold.go: DDSketchIsotropicThreshold (T=ε·N/(k·B)), alongside the existing CountSketch F2IsotropicThreshold; returns +Inf for B==0 so cold start (N=0,B=0) falls through to the caller's floor-at-1 (ship-on-first-touch) — "cold start is a feature". - sketches/ddsketch.go (DDSketchWrapper): SetGosMode, GosDeltaThreshold, a dirty-list + one-shot wake signal (recordDirty/ConsumeWakeSignal), and drainGosDelta wired into ComputeDeltaAgainst — the GOS branch drains the insert-time dirty list into a proto DDSketchDelta and never consults prev (no snapshot decode). Update() routes through sk.UpdateGOS when gosEpsilon>0. Reset clears the per-window dirty/wake state (gosEpsilon/gosSites are per-series config, survive). - precompute.go: applyGosMode is already generic (structural SetGosMode assert); extended subWindowShouldEmit and subWindowMarkEmitted with a SketchTypeDDSketch branch mirroring CountSketch (bypass Gate-1's divergence pre-check in GOS mode). - asapedgeprocessor warm_sketch.go: the DDSketch factory now primes SetGosMode at series CREATION (like the plain CountSketch factory) so inserts before a new series' first flush are gated too; the generic opts.gosDeltaEpsilon/gosSites plumbing and subWindowEnabled() already cover DDSketch unchanged. - asapedgeprocessor config_validate.go / config.go: gos_delta_epsilon is now accepted for family=ddsketch as well as non-heap countsketch, and rejected (with a clear message) on every other family. Tests (all three tiers, mirroring CountSketch): - sketchlib-go DDSketch (in the sibling sketchlib-go commit): disabled-passthrough, deterministic crossing+reset, first-touch-only B tracking, and telescoping reconstruction (exact bucket parity). - asap-precompute-go/sketches ddsketch_gos_test.go: empty-drain-returns- nil, wake fires once per batch and clears on read, wrapper-level telescoping via ComputeDeltaAgainst+ApplyDelta (+ residual) matching a never-reset reference in count and quantiles. - asapedgeprocessor gos_ddsketch_test.go: TestGOSDDSketchInsertWakesFlush — full pipeline, SubWindowInterval UNSET and WindowDuration=1h (neither can fire on its own), DDSketch observations through ConsumeMetrics alone produce a flushed envelope with no manual wakeSubWindow call. gos_ddsketch_validate_test.go: per-family accept/reject matrix. go build / go vet / go test clean on sketchlib-go's DDSketch package, asap-precompute-go (all subpackages), and asapedgeprocessor. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
zzylol
force-pushed
the
split/pr-gos-ddsketch
branch
from
July 17, 2026 02:35
3a18b97 to
250d7a8
Compare
zzylol
changed the base branch from
split/pr-gos-countsketch
to
split/pr-gos-cms
July 17, 2026 02:35
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the GOS per-family stack (base: CountSketch
split/pr-gos-countsketch/ #524). Depends on sketchlib-go #75 (sparse full-snapshot wire fix +UpdateGOS/PopulatedBuckets).Converts DDSketch's delta gating to insert-time detect+reset (design §11). Formula
T=εN/(kB)(derivations §8.4), withB(populated-bucket count) tracked EXACTLY via first-touch increment / reset decrement — never assumed, since an assumedB<B_actualdoesn't just loosen but VIOLATES the staleness bound.DDSketchWrapper:SetGosMode,DDSketchIsotropicThreshold, dirty-list + one-shot wake,drainGosDeltawired intoComputeDeltaAgainst;SketchTypeDDSketchbranches insubWindowShouldEmit/subWindowMarkEmitted; factory priming;gos_delta_epsilonaccepted forfamily=ddsketch.Known interaction with sampling PR #518 (flagged for #518's reattachment, not a blocker here): #518 adds a row-admission path (
ApplyAdmittedOccurrence) that bypasses GOS detection — a DDSketch with both sampling andgos_delta_epsilon>0would silently skip insert-time detection. Composing them is #518's job when it rebases onto this stack; estimates stay unbiased.Cross-repo follow-up (from the sketchlib-go half): the Rust backend (ASAPQuery-backend) needs a decoder update for the new sparse
bucketsproto field before it can read sparse-encoded outlier sketches (until then they decode as empty there — still strictly better than a multi-GB dense alloc).Test plan
TestGOSDDSketchInsertWakesFlush+ per-family validation matrix.🤖 Generated with Claude Code