fix(gorilla-merger): route old cold-only queries to the merger StoreAPI - #341
Merged
Merged
Conversation
The decode-on-read cold path stored parts but served EMPTY for old windows: customStore.timeRange() advertised MinTime = tsdb StartTime (roughly the recent head min), ignoring cold coverage. thanos-query prunes a store whose advertised range does not overlap the query, so a query for old (cold-only) data was never routed to the merger and streamColdSeries never ran. Lower the advertised MinTime to the oldest cold part's block_start_ms when a cold querier is attached (via ColdPartStore.MinBlockStart / ColdQuerier.MinBlockStart), so the StoreInfo + TsdbInfos cover the cold window and thanos-query fans the query out to the merger. Add debug logs at streamColdSeries entry and emit so cold-serve activity is observable. Tests: MinBlockStart accessor, timeRange folds in cold coverage, and an end-to-end Series RPC over an old cold-only window returns the stored part's series + samples. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
zzylol
added a commit
that referenced
this pull request
May 26, 2026
…ld reload so cold queries serve (#342) Cold parts were stored but a live cold query still returned empty after a restart, even with #341's MinTime floor in place. Two compounding causes: 1. customStore.timeRange() advertised MaxInt64 for an EMPTY tsdb head: tsdb.DB.StartTime() returns math.MaxInt64 when the head holds no samples (e.g. right after a restart, before the first warm fragment lands). That sentinel leaked into the StoreAPI's advertised MinTime, so thanos-query pruned the merger from EVERY query (no window can be >= MaxInt64) and streamColdSeries never ran. Take the MIN of the tsdb StartTime and the cold-part floor, and never advertise MaxInt64 (fall back to MinInt64 when the store is genuinely empty so it stays discoverable, returning no series). 2. ColdPartStore.Reload ran synchronously in main before the StoreAPI/HTTP servers started. It fetches + OpenParts every stored part, so a large accumulated cold tier (thousands of parts) blocked startup for tens of seconds — during which the gRPC endpoint was down and every query (warm and cold) returned empty. Run the reload in the background; the manifest is mutex-guarded, so concurrent queries see a growing manifest until it completes and the warm/open path serves immediately. Live: after a restart the StoreAPI now comes up ~80ms after WAL replay (vs ~53s before) and warm queries return data while the cold reload runs in the background; cold-only-window queries return the stored parts' series once reload completes (streamColdSeries emits N series). Adds a regression test asserting the empty-head MinTime never advertises MaxInt64 and drops to the cold floor when parts exist. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
customStore.timeRange()(gorilla-mergerinternal/merger/customstore.go) advertisedMinTime = tsdb StartTime(~the recent head min), ignoring cold-part coverage. thanos-query prunes a store from a query's fan-out when the query window falls below the store's advertisedMinTime(used inStoreInfo+TsdbInfos). So a query for old, cold-only data was never routed to the merger,streamColdSeriesnever ran, and storedcold/*.partobjects were served EMPTY — even though the write path worked.MinTimeto the oldest cold part'sblock_start_ms(newColdPartStore.MinBlockStart/ColdQuerier.MinBlockStart). The per-series[min_ts,max_ts]index entries remain the authoritative time filter inside the query path; this only widens the advertised floor of what the merger might serve.streamColdSeriesentry and emit (cold_seriescount) so cold-serve activity is observable.Why the existing union test passed but live failed
TestStoreAPIUnionsColdAndTSDBcallscustomStore.Seriesdirectly, bypassing thanos-query's Info-based store pruning. Live, thanos-query consults the advertised time range first and never dispatched the RPC — exactly the wiring/time-units class of bug the task flagged.Test plan
go build ./...,go vet ./...,gofmt -lcleango test ./...green (existing tests unbroken)TestColdPartStoreMinBlockStart— min block start across out-of-order parts; empty/nil guardsTestCustomStoreTimeRangeIncludesCold— tsdb-only min = StartTime; with cold attached, min drops to the old cold block startTestCustomStoreSeriesServesOldColdWindow— end-to-endSeriesRPC over a window covering ONLY an old cold part returns the stored series' labels + samples🤖 Generated with Claude Code