Conversation
…ropic)
Adds the GOS (Geometric-OctoSketch) norm-adaptive relative delta gate as
an opt-in replacement for the fixed DeltaThreshold, Count-Sketch families
only:
- Isotropic (default): one scalar T = ε‖Ĉ‖/(2k√(dw)) for all cells,
O(1) edge memory.
- Anisotropic (opt-in via gos_anisotropic): gradient-weighted per-cell
{T_j} water-filling, less communication on skewed sketches at
O(d·w) edge memory for the threshold vector.
Wired end-to-end: PrecomputeConfig.Gos{DeltaEpsilon,Sites,Anisotropic} ->
CountSketchWrapper.SetGosMode -> ComputeDeltaAgainst's threshold override
(precompute.go's applyGosMode calls this per emit) -> the new
gos_delta_epsilon/gos_sites/gos_anisotropic control-plane knobs on
asapedgeprocessor's MetricFamily config.
Part of splitting #515 into a reviewable stack (cleanup -> GOS ->
NitroSketch sampling -> eval/docs). Builds/vets/tests clean standalone
(asap-precompute-go + asapedgeprocessor).
This was referenced Jul 16, 2026
…rams) The anisotropic per-cell water-filling mode depends on an Activity_j input (|current-prev|) defined against a single, uniformly-timed snapshot. Under the insert-time reset-on-send transport redesign (see the docs commit that follows), different cells can now reset at different times, and the old Activity definition is no longer well-defined without reintroducing a periodic full-matrix snapshot — this needs its own redesign (a per-cell EMA of |delta| is the leading candidate, but hasn't been checked against the water-filling's error guarantee). Rather than ship known-stale math, remove the anisotropic code path entirely: AllocateThresholds/GosCell/GosParams, CountSketchWrapper.computeAnisotropicDelta/gosThresholdMatrix, the gosAnisotropic field and SetGosMode's anisotropic parameter, and the GosAnisotropic config knob end-to-end (asap-precompute-go's PrecomputeConfig, asapedgeprocessor's MetricFamily/sketchOpts/ processor.go wiring). The isotropic path (F2IsotropicThreshold, ComputeDeltaAgainst's threshold override) is untouched. Tests: gos_eval_test.go (100% anisotropic, TestGosAnisoSavingsRatio) deleted; gos_threshold_test.go trimmed to the isotropic-only TestGosModeToggle + TestF2IsotropicClosedForm; gos_config_test.go's TestGosKnobsDecode drops the gos_anisotropic assertions, keeps gos_delta_epsilon/gos_sites.
design-gos-unified-edge-telemetry.md: new §11 "The 2026-07 redesign: no sub-window, insert-time detection for all 6 families" — the unified model (check-at-insert, sync-only, alerting moves to the backend), the wake-on-demand flush mechanism that keeps the existing OTLP/ SketchEnvelope pipeline (only the trigger changes, from purely-timer to timer-or-woken), retiring Gate 1 (subWindowShouldEmit/ackedCells) and CMS's local point-query read, a per-family cell/threshold/reset mechanism table, why the cold-start "threshold near zero" behavior is intentional, and three explicitly-scoped open items (anisotropic Activity_j redesign, DDSketch's unbounded bucket growth — sketchlib-go#72, independent of this work — and HLL's unverified small-cardinality regime). Old §11/§12 renumbered to §12/§13, with a cross-reference note where the old "acked snapshot" description is superseded. sampling-cdm-gos-derivations.md §8: fills in the previously-empty CDM staleness/threshold derivations for CMS (§8.2, L1/max-composition, T=εN/k — no √(dw), unlike CountSketch's L2/sum-composition), DDSketch range counts (§8.4, L1/sum-composition, T=εN/(kB) with B tracked O(1) per new bucket rather than assumed — an assumed constant would let the staleness bound be silently violated, not just loosened), KLL's concrete emit trigger (§8.6, R>=εN from the existing R/N bound), and HLL's register-change adapter (§8.7, OctoSketch's own max-merge treatment: |2^C'-2^C|>=2^tau on the linearized value, never reset — MAX-merge is idempotent). §11 summary table gets an "Isotropic threshold" column tying all 6 families together; §13 implementation notes updated to match (sub-window language replaced, CMS point-query retirement noted).
zzylol
added a commit
that referenced
this pull request
Jul 16, 2026
Evaluation for the GOS + NitroSketch sampling work in this stack: - DEBS 2022 real-backend query-accuracy harness (all 4 sketch query types), Google cluster trace integrated ε-sweep (accuracy·latency·freshness·resources, single-node + cluster), C1 bandwidth/encoding-factor experiments (DDSketch/HLL vs raw+gzip), anisotropic-vs-isotropic per-cell delta measurement (historical — anisotropic itself was later removed from #517 pending an Activity_j redesign; this eval data predates that and is kept for the record), raw-vs-sketch cardinality crossover. - otel_collector_benchmark/epsilon_floor: ε-floor vs NitroSketch benchmark, in-tree. - docs/gos-eval-results.md, docs/phase-2.md: recorded eval results and phase-2 implementation history. docs/design-gos-unified-edge-telemetry.md and docs/sampling-cdm-gos-derivations.md moved to #517 (the GOS PR they actually describe) instead of living here. Part of splitting #515 into a reviewable stack (cleanup -> GOS -> NitroSketch sampling -> eval/docs) — the last PR in the stack.
This was referenced Jul 16, 2026
Contributor
Author
|
Superseded by the 6-family GOS insert-time redesign (design-gos-unified-edge-telemetry.md §11), all merged into main via #524→#525→#528→#526→#527→#529→#530→#531→#532. This PR's CountSketch-only isotropic threshold ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second PR in the split of #515 (cleanup → GOS → NitroSketch sampling → eval/docs). Base: #516.
Adds the GOS (Geometric-OctoSketch) norm-adaptive relative delta gate as an opt-in replacement for the fixed
DeltaThreshold, Count-Sketch families only — isotropic threshold: one scalarT = ε‖Ĉ‖/(2k√(dw))for all cells, O(1) edge memory.Wired end-to-end:
PrecomputeConfig.Gos{DeltaEpsilon,Sites}→CountSketchWrapper.SetGosMode→ComputeDeltaAgainst's threshold override (precompute.go'sapplyGosModecalls this per emit) → thegos_delta_epsilon/gos_sitescontrol-plane knobs onasapedgeprocessor'sMetricFamilyconfig.Anisotropic mode removed (was originally shipped here, then pulled back out): the per-cell water-filling's
Activity_jinput depends on a single, uniformly-timedprevsnapshot, which the design docs added in this PR (§11 ofdesign-gos-unified-edge-telemetry.md) make obsolete for the isotropic path — an insert-time, reset-on-send transport where different cells reset at different times. Reintroducing a fair Activity definition under that model is an open design problem (a per-cell EMA of|Δ|is the leading candidate, unverified against the water-filling's error guarantee), soAllocateThresholds/GosCell/GosParamsand the anisotropic code path were deleted rather than shipping known-stale math. Tracked as an explicit TODO in the design doc.New docs (moved here from what was originally planned for #519, since they describe this PR's own algorithm):
docs/design-gos-unified-edge-telemetry.md§11 (the unified no-sub-window, insert-time-detection transport design covering all 6 sketch families — Sum/CMS/CountSketch/DDSketch/KLL/HLL) anddocs/sampling-cdm-gos-derivations.md§8 (the per-family CDM staleness/threshold math: CMS'sL1/max-compositionT=εN/k, DDSketch'sL1/sum-compositionT=εN/(kB), KLL'sR≥εNemit trigger, HLL's OctoSketch-paper register-change adapter|2^C'-2^C|≥2^τ).Two files (
asapedgeprocessor/config.go,warm_sketch.go) also carry unrelated NitroSketch-sampling changes in the full branch — only the GOS-specific hunks are in this PR; the rest land in PR 3 (#518).Test plan
asap-precompute-goandasapedgeprocessorbuild/vet/test clean standalone on this branch (isotropic-only, post anisotropic removal).🤖 Generated with Claude Code