docs(design): holistic edge→backend compression (cold chunk + warm sketch) - #331
Merged
Merged
Conversation
…etch)
Design (for review, not code) for a custom lossless cold chunk format and a
unified offset/frame-of-reference scheme that also shrinks warm sketch state.
- Cold chunk: best-of-N {Gorilla-XOR, INT_FOR_DELTA, INT_FOR_DOD}, all lossless,
with a decimal-exactness guard; write path stores to S3 WITHOUT decode; a
decode-on-read Thanos StoreAPI decodes to XOR at query time so a stock PromQL
engine queries the compact S3 form.
- Warm sketch: per-family offset/FOR/delta/sparse (SUM/KLL use the offset;
DDSketch index-FOR; HLL sparse; CMS narrow-counter), plus offset×delta-
transmission with the offset bound to the Full-snapshot epoch.
- offset-drift → Full re-base, unifying warm (re-emit Full) and cold (cut chunk,
new base); per-series adaptive Full cadence + heartbeat; counters use delta.
Informed by an offline benchmark on the Chimp/Serf real datasets: INT FOR+delta
beats Gorilla ~4.8x on fixed-decimal series (the majority); Gorilla wins on true
high-precision floats — hence best-of-N. Decided out of scope: no zstd, no lossy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend stores the wire bytes opaquely (no re-serialize / no part recompression), so wire = sketch_db storage = disk parts — one serialize change wins on all three. Measured (N=5000): HLL dense 16,532 B flat → sparse 5–50× (P1, also cuts warm memory since stored uncompressed); KLL raw-f64 2,157 B → (v−offset) fixed-point ~2× (P2). DDSketch (556 B, already FOR+varint), CMS/CountSketch (~12.5 KB, already zigzag-varint), SUM (OTLP-framing-bound) already well-encoded — skip. Supersedes the earlier DDSketch-index-FOR / narrow-CMS rows. §5.2 resolved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Measured (/mydata/xseries-bench, real cluster groups + synthetic): shared timestamp column is the cross-series win — ts is 36-52% of per-series bytes (values FOR-compress well), so sharing it across a k-series group saves -43% on the real aggregate, correlation-independent (the Heracles result). The cross-series VALUE base is net-NEGATIVE (-3.5%): predictor is noise/signal ratio not correlation, per-series delta already took the common bits, plus a float-domain trap. Adopt shared-ts grouped layout; cross-series value base is opt-in only for near-replica groups. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v1 is zero-tuning: drift = hard/overflow only (re-base when a residual exceeds the frame's chosen integer width — a correctness bound, not a knob); heartbeat = fixed N windows (= existing Full cadence; cold's chunk bound already serves). Counters use delta + heartbeat. The soft/efficiency-K drift (the only tunable) is deferred to a later optimization, added only if long-lived frames degrade without overflowing. §5.1 marked decided. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sampling-enhanced sketches compose with the compression scheme on orthogonal, complementary axes: compression cuts bytes, sampling cuts ingest CPU/update-rate (the axis FOR/offset can't touch — it's what reduces hashing/compaction cost). Warm-only (cold raw stays lossless). Composition rule: store the RAW SAMPLED integer state + a global p per frame, rescale ×1/p at QUERY (not store), so varint/FOR/sparse keep working on small integers. Per-family: HLL hash-threshold sampling synergizes strongly with sparse encoding (thins registers, extends the sparse regime ~1/p×); DDSketch/CMS store smaller sampled counts; KLL weighted insert is orthogonal. Error budgets add (per-family bounds, KLL pN≳k²); p is a per-metric control-plane knob alongside tier/sketch-type. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…et constant The cold (VM/INT_FOR) and warm (KLL/SUM) paths share the parse-once computation — the per-series decimal SCALE (must be identical so both land in the same integer domain) + value stats — but NOT a single offset constant: cold re-bases per block (tightest per-block width) while warm re-bases per Full-epoch (stable base so Deltas stay mergeable); those cadences conflict. Scope is the raw-value- offset families only (cold-INT / KLL / SUM); DDSketch (index-FOR), HLL (hash), CMS (counters) have no shared value offset. Kind matches per shape (gauge→FOR, counter→delta), not as a number. Also note sampling (§6) is what closes the sketch-build CPU gap the offset can't. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Formal backing for §6's cited bounds: the four sampling-enhanced sketches with
their algorithms + unbiasedness/variance/concentration proofs and final boxed
bounds — DDSketch (inverse-prob bucket; x_q∈(1±α)x_{q±ε_s}), KLL (weighted
insert; ε_k+ε_s, pN≳k²), HLL (hash-threshold, not additive; RSE≈√((1-p)/(pn)+
1.04²/m); why per-occurrence is biased), Nitro/CMS/CountSketch (additive; var
(1-p)/p·Σa²; CM loses no-underestimate). Design rules by update type. §6.5 (the
Go benchmark proving these empirically vs sketchlib-go) lands separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…g benchmark Measured vs real sketchlib-go (harness /mydata/sampling-bench): all §7 bounds held. p=0.1 → ~10× fewer updates + ~6× wall-clock per family. KLL pN≳k² confirmed as a hard boundary; HLL hash-threshold is unbiased to p=0.01 (n≥1e5) while per-occurrence blows up (0.66→3.8) — and the threshold hash MUST be independent of HLL's register hash (else 15–84% bias). Sampled serialization never bloats (composition rule holds). Caveats: HLL m is compile-time const; sparser-HLL win only below register saturation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Geometric skip-ahead is a statistically-identical, cheaper implementation of per-update Bernoulli(p) (gaps ~Geometric(p) ⟺ iid keep w.p. p): same §7 bounds / composition rule, amortizes sampling RNG to ~O(p)/item. Applies to the per-update families (DDSketch/KLL/CMS/CountSketch/Nitro), NOT HLL (hash-threshold per key, already RNG-free). Orthogonal to shared-ts and the offset — it touches neither timestamps nor the value base. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 phases, ordered by isolation+ROI+dependency+risk. Parallel start set PR1 (HLL-sparse) ∥ PR2 (KLL-offset) ∥ PR5 (cold INT codec lib); critical path PR5→PR6(StoreAPI)→PR7(shared-ts). Sampling (PR3 gated no-op + PR4 controller p) in phase 2; offset unification + warm drift→Full in phase 4. Format-changing PRs use additive proto fields + dual-read for staggered rollout. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… measured) The 4.8× came from VictoriaMetrics lib/encoding WITH zstd-wrapping; since we decided no-zstd (§5), the realized win is the no-zstd FOR+delta codec measured in the intchunk PR: ~2.2-3.6× on fixed-decimal. A varint-residual refinement is being added to push toward ~3×+ without zstd. §0 + §8 updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…#434) Varint-residual sub-codecs landed in the intchunk best-of-N; on the Chimp/Serf fixed-decimal class the aggregate moved 2.22x -> 2.33x vs Gorilla (up to ~3.6x per series). Bottom line: no-zstd cold compression caps ~2.3x; the 4.8x needs zstd (excluded). Gorilla-XOR stays the lossless fallback for true floats. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design doc for review (no code) —
docs/design/holistic-edge-backend-compression.md.What it specs
AggrChunks at query time, so S3 holds the compact custom form while a stock PromQL engine queries it.Evidence
Offline benchmark on the Chimp/Serf real datasets (
/mydata/compress-bench): INT FOR+delta beats Gorilla ~4.8× (up to ~10×) on fixed-decimal series (the majority of real metrics); Gorilla wins on true high-precision float → hence the best-of-N.Decided out of scope
No zstd-wrapped variants, no lossy/Serf — cold stays purely lossless.
Open decisions (§5)