docs(gos): unified isotropic-threshold design + per-family derivations - #520
Merged
Merged
Conversation
design-gos-unified-edge-telemetry.md: new §11 "The 2026-07 redesign: no sub-window, insert-time detection for all 6 families" — the unified model (check-at-insert, sync-only, alerting moves to the backend), the wake-on-demand flush mechanism that keeps the existing OTLP/ SketchEnvelope pipeline (only the trigger changes, from purely-timer to timer-or-woken), retiring Gate 1 (subWindowShouldEmit/ackedCells) and CMS's local point-query read, a per-family cell/threshold/reset mechanism table, why the cold-start "threshold near zero" behavior is intentional, and three explicitly-scoped open items (anisotropic Activity_j redesign, DDSketch's unbounded bucket growth — sketchlib-go#72, independent of this work — and HLL's unverified small-cardinality regime). Old §11/§12 renumbered to §12/§13, with a cross-reference note where the old "acked snapshot" description is superseded. sampling-cdm-gos-derivations.md §8: fills in the previously-empty CDM staleness/threshold derivations for CMS (§8.2, L1/max-composition, T=εN/k — no √(dw), unlike CountSketch's L2/sum-composition), DDSketch range counts (§8.4, L1/sum-composition, T=εN/(kB) with B tracked O(1) per new bucket rather than assumed — an assumed constant would let the staleness bound be silently violated, not just loosened), KLL's concrete emit trigger (§8.6, R>=εN from the existing R/N bound), and HLL's register-change adapter (§8.7, OctoSketch's own max-merge treatment: |2^C'-2^C|>=2^tau on the linearized value, never reset — MAX-merge is idempotent). §11 summary table gets an "Isotropic threshold" column tying all 6 families together; §13 implementation notes updated to match (sub-window language replaced, CMS point-query retirement noted).
Cut the "current vs target" sampling-rollout table (§3.1), the redundant §3.2 "Status." paragraph (which had drifted out of sync with §3.1's own later claim), and §12's "Implementation status as of this writing" block and half of "Ties to existing code" — several of which pointed at code already deleted (AllocateThresholds/gos_threshold matrix, gos_anisotropic, f2_coord.rs/f2engine.go from the F2 removal). Net: shorter, and no longer self-contradictory or pointing at dead files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3 tasks
Fold the former standalone "2026-07 redesign" section into the rest of the document instead of leaving it as a dated patch note bolted on at the end: - The unified insert-time model, wake-on-demand flush, per-family detection/reset table, and cold-start rationale move into §7 as "Layer D" (Layers A-C already establish *what* threshold each cell gets; Layer D is *when* a crossing is detected/shipped) - the same section that already derives the closed-form thresholds these checks use, instead of sitting in a separate, differently-scoped section. - The retirement specifics (Gate 1, CMS point-query, Discipline B alerting) move to §11 (Implementation notes), next to the existing "Ties to existing code" - this is where migration-from-current-code detail already lives, so it no longer needs a "superseded by §11" cross-reference note explaining itself. - The open items (Activity_j redefinition, DDSketch bucket growth, HLL small-cardinality) merge into the pre-existing Open problems list instead of duplicating it as a second list. Sections renumber 11->omitted, 12->11, 13->12; no content dropped, only relocated and de-dated. Net -9 lines despite the merge, from removing the now-unneeded cross-reference scaffolding. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First PR in an 11-PR GOS stack (docs → transport → per-family GOS × 6, interleaved with 2 retirement PRs and a DDSketch wire-format PR). Base: #516.
This is design/spec content, split out to stand on its own rather than being bundled with any single family's implementation:
docs/design-gos-unified-edge-telemetry.md§11: the unified no-sub-window, insert-time-detection model for all 6 sketch families (Sum/CMS/CountSketch/DDSketch/KLL/HLL) — wake-on-demand flush mechanics, Gate 1 retirement rationale, Discipline B split rationale, per-family cell/threshold/reset table, why cold-start behavior is intentional, and explicitly-scoped open items (anisotropic Activity_j redesign, DDSketch's unbounded bucket growth, HLL's small-cardinality regime).docs/sampling-cdm-gos-derivations.md§8: the per-family CDM staleness/threshold math — CMS'sL1/max-compositionT=εN/k, DDSketch'sL1/sum-compositionT=εN/(kB), KLL'sR≥εNtrigger, HLL's OctoSketch-paper register-change adapter|2^C'-2^C|≥2^τ.Note: sketchlib-go already has an
octosketchworker/aggregator framework with a similar algorithmic shape, but it's built for intra-process multi-core parallelism (in-memory Go channels, queue-depth-adaptive threshold) rather than edge→backend network transmission with an accuracy-driven (ε-relative) threshold — the implementation PRs that follow build a purpose-built mechanism rather than reusing it.Test plan
🤖 Generated with Claude Code