perf(asapedgeprocessor): accumulate intchunk cold parts per block_duration - #442
Merged
Merged
Conversation
…ation The intchunk cold-part path shipped one coldpart.Part per flush/drain, and each per-shard drain carries only ~1-2 samples/series (the gorilla fragment encoder emits XOR-chunk fragments well inside the window, and the cold path re-encoded + POSTed each emission immediately). A Part stores a per-series index + a deduped label symbol table, so at that density the fixed per-part/per-series overhead dominates: the whole part measured ~3x LARGER than the equivalent gorilla-XOR even though the intchunk value codec itself is ~1.5x smaller on the same samples. Add a per-shard cold-part accumulator that buffers each flush's drained fragments (decoded + merged by series) and seals + POSTs ONE part only once the buffered absolute-ms sample span reaches cold.block_duration (default = window_duration, 60s) — amortizing the index/symtab overhead over ~a block's worth of samples/series and flipping the realized ratio to smaller than gorilla. Shutdown force-seals any partial sub-block tail so no cold samples are lost. The default fragment path and the merger /ingest/coldpart contract (a complete part, absolute-ms block bounds) are unchanged; this is intchunk-cold-part-only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
coldpart.Partper flush/drain. Each per-shard drain carries only ~1-2 samples/series (the gorillaStreamingFragmentEncoderemits XOR-chunk fragments well inside its bounded-OOO window, and the cold path re-encoded + POSTed each emission immediately). APartstores a per-series index + a deduped label symbol table, so at that sample density the fixed per-part/per-series overhead dominates — the whole part measured ~3× LARGER than the equivalent gorilla-XOR (0.32×), even though the intchunk value codec itself is ~1.5× smaller on the same samples.coldPartAccumulator) that buffers each flush's drained fragments (decoded + merged by series, dedup on exact-T) and seals + POSTs ONE part only once the buffered absolute-ms sample span reachescold.block_duration(default =window_duration, 60s). This amortizes the index/symtab overhead over ~a block's worth of samples/series, flipping the realized ratio to ~1.5–2.3× SMALLER than gorilla./ingest/coldpartcontract is unchanged — still a completecoldpart.Partwith absolute-ms block bounds. The buffering is gated oncold.format == intchunk.Why parts were ~1s
The cold path sealed a part on every flush/drain (
shipColdPart), and a single drain'sfragmentsToSeriesblock bounds = min/max sample T across just that drain's ~1-2 samples/series. So a part's time span was the span of one flush emission, not the configured window. Accumulating across flushes fixes the granularity at the source.block_duration note
block_durationdefaults towindow_duration(60s). At the observed ~1.8 samples/series per second, a 60s block yields ~100 samples/series, which is enough to amortize the per-series index + symtab. If a deployment's effective samples/series/block stays low (sparse series, short scrape bursts), raisingblock_durationabove 60s further improves amortization at the cost of cold-tier delivery latency.Test plan
go build ./...clean (asapedgeprocessor)go vet ./...cleangofmt -lclean on changed filesTestColdPartAccumulatesAcrossFlushes: 3 in-block flushes (4 samples/series each) emit 0 parts on flush and exactly 1 part on Shutdown, covering the union (Σ = 12 samples/series, 2 series) with block bounds = [first flush sample, last flush sample].TestColdPartSealsAtBlockBoundary: a flush whose accumulated span crossesblock_durationseals a part without waiting for Shutdown.go test ./...green (existing cold/fragment/sketch/stagger tests unaffected).🤖 Generated with Claude Code