Skip to content

perf(asapedgeprocessor): accumulate intchunk cold parts per block_duration - #442

Merged
zzylol merged 1 commit into
mainfrom
feat/cold-part-block-accum
May 26, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/cold-part-block-accum

Conversation

@zzylol

@zzylol zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • The intchunk cold-part path POSTed one coldpart.Part per flush/drain. Each per-shard drain carries only ~1-2 samples/series (the gorilla StreamingFragmentEncoder emits XOR-chunk fragments well inside its bounded-OOO window, and the cold path re-encoded + POSTed each emission immediately). A Part stores a per-series index + a deduped label symbol table, so at that sample density the fixed per-part/per-series overhead dominates — the whole part measured ~3× LARGER than the equivalent gorilla-XOR (0.32×), even though the intchunk value codec itself is ~1.5× smaller on the same samples.
  • Add a per-shard cold-part accumulator (coldPartAccumulator) that buffers each flush's drained fragments (decoded + merged by series, dedup on exact-T) and seals + POSTs ONE part only once the buffered absolute-ms sample span reaches cold.block_duration (default = window_duration, 60s). This amortizes the index/symtab overhead over ~a block's worth of samples/series, flipping the realized ratio to ~1.5–2.3× SMALLER than gorilla.
  • Shutdown force-seals any partial (sub-block) tail synchronously under the shutdown deadline so no cold samples are lost.
  • The default fragment path is untouched (the per-shard router falls straight through to the existing async ship worker), and the merger /ingest/coldpart contract is unchanged — still a complete coldpart.Part with absolute-ms block bounds. The buffering is gated on cold.format == intchunk.

Why parts were ~1s

The cold path sealed a part on every flush/drain (shipColdPart), and a single drain's fragmentsToSeries block bounds = min/max sample T across just that drain's ~1-2 samples/series. So a part's time span was the span of one flush emission, not the configured window. Accumulating across flushes fixes the granularity at the source.

block_duration note

block_duration defaults to window_duration (60s). At the observed ~1.8 samples/series per second, a 60s block yields ~100 samples/series, which is enough to amortize the per-series index + symtab. If a deployment's effective samples/series/block stays low (sparse series, short scrape bursts), raising block_duration above 60s further improves amortization at the cost of cold-tier delivery latency.

Test plan

  • go build ./... clean (asapedgeprocessor)
  • go vet ./... clean
  • gofmt -l clean on changed files
  • New TestColdPartAccumulatesAcrossFlushes: 3 in-block flushes (4 samples/series each) emit 0 parts on flush and exactly 1 part on Shutdown, covering the union (Σ = 12 samples/series, 2 series) with block bounds = [first flush sample, last flush sample].
  • New TestColdPartSealsAtBlockBoundary: a flush whose accumulated span crosses block_duration seals a part without waiting for Shutdown.
  • Full package go test ./... green (existing cold/fragment/sketch/stagger tests unaffected).

🤖 Generated with Claude Code

…ation

The intchunk cold-part path shipped one coldpart.Part per flush/drain, and
each per-shard drain carries only ~1-2 samples/series (the gorilla fragment
encoder emits XOR-chunk fragments well inside the window, and the cold path
re-encoded + POSTed each emission immediately). A Part stores a per-series
index + a deduped label symbol table, so at that density the fixed
per-part/per-series overhead dominates: the whole part measured ~3x LARGER
than the equivalent gorilla-XOR even though the intchunk value codec itself is
~1.5x smaller on the same samples.

Add a per-shard cold-part accumulator that buffers each flush's drained
fragments (decoded + merged by series) and seals + POSTs ONE part only once
the buffered absolute-ms sample span reaches cold.block_duration (default =
window_duration, 60s) — amortizing the index/symtab overhead over ~a block's
worth of samples/series and flipping the realized ratio to smaller than
gorilla. Shutdown force-seals any partial sub-block tail so no cold samples
are lost. The default fragment path and the merger /ingest/coldpart contract
(a complete part, absolute-ms block bounds) are unchanged; this is
intchunk-cold-part-only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant