Skip to content

fix(gorilla-merger): serve cold queries after restart (empty-head MinTime + non-blocking reload) - #342

Merged
zzylol merged 1 commit into
mainfrom
fix-cold-query-empty
May 26, 2026
Merged

zzylol merged 1 commit into
mainfrom
fix-cold-query-empty

Conversation

@zzylol

@zzylol zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor

Summary

Cold parts were stored but a live cold query still returned empty after a merger restart, even with #341's MinTime floor in place and streamColdSeries debug logs added. Investigation on the live multinode cluster (node1) found TWO compounding root causes, neither addressed by #341:

  1. customStore.timeRange() advertised math.MaxInt64 for an EMPTY tsdb head. tsdb.DB.StartTime() returns MaxInt64 when the head holds no samples (verified empirically) — exactly the state right after a restart, before the first warm fragment lands, while cold parts already exist in S3. That sentinel leaked into the StoreAPI's advertised MinTime, so thanos-query pruned the merger from every query (no window can be >= MaxInt64) and streamColdSeries never ran — the served-empty symptom. Fix: advertise the MIN of the tsdb StartTime and the cold-part floor, and never advertise MaxInt64 (fall back to MinInt64 when the store is genuinely empty so it stays discoverable and returns no series cheaply).

  2. ColdPartStore.Reload blocked StoreAPI startup. It ran synchronously in main() before the gRPC/HTTP servers started, fetching + OpenPart-ing every stored part. With a large accumulated cold tier (live: ~2000 parts), this took ~53s during which the merger served nothing (warm or cold). Fix: run the reload in the background; the manifest is mutex-guarded so concurrent queries see a growing manifest until it completes, and the warm/open path serves immediately.

Live demonstration (node1)

old image (#341) fixed image
StoreAPI up after WAL replay ~53s (blocked on Reload) ~80ms
warm query during cold reload EMPTY returns data
cold-only-window query (post-reload) n/a series=1, streamColdSeries: emitting cold series cold_series=476

After restart with the fixed image: gorilla-merger up logged ~80ms after WAL replay while reloaded cold manifest (2014 parts) was still running in the background; warm count(http_requests_total) returned data immediately; a cold-only deep-window query_range returned the stored parts' series and the merger logged streamColdSeries: emitting cold series cold_series=476.

Test plan

  • go build ./..., go vet ./..., gofmt -l clean
  • go test ./... green
  • New regression TestCustomStoreTimeRangeEmptyHead FAILS without the timeRange fix (empty head advertised MinTime = MaxInt64; merger would be pruned) and PASSES with it
  • Live: cold-only query returns data + streamColdSeries invoked after a restart on the multinode cluster

🤖 Generated with Claude Code

…Time + non-blocking reload)

Cold parts were stored but a live cold query still returned empty after a
restart, even with #341's MinTime floor in place. Two compounding causes:

1. customStore.timeRange() advertised MaxInt64 for an EMPTY tsdb head:
   tsdb.DB.StartTime() returns math.MaxInt64 when the head holds no samples
   (e.g. right after a restart, before the first warm fragment lands). That
   sentinel leaked into the StoreAPI's advertised MinTime, so thanos-query
   pruned the merger from EVERY query (no window can be >= MaxInt64) and
   streamColdSeries never ran. Take the MIN of the tsdb StartTime and the
   cold-part floor, and never advertise MaxInt64 (fall back to MinInt64 when
   the store is genuinely empty so it stays discoverable, returning no series).

2. ColdPartStore.Reload ran synchronously in main before the StoreAPI/HTTP
   servers started. It fetches + OpenParts every stored part, so a large
   accumulated cold tier (thousands of parts) blocked startup for tens of
   seconds — during which the gRPC endpoint was down and every query (warm
   and cold) returned empty. Run the reload in the background; the manifest
   is mutex-guarded, so concurrent queries see a growing manifest until it
   completes and the warm/open path serves immediately.

Live: after a restart the StoreAPI now comes up ~80ms after WAL replay
(vs ~53s before) and warm queries return data while the cold reload runs in
the background; cold-only-window queries return the stored parts' series once
reload completes (streamColdSeries emits N series). Adds a regression test
asserting the empty-head MinTime never advertises MaxInt64 and drops to the
cold floor when parts exist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 40334e2 into main May 26, 2026
@zzylol
zzylol deleted the fix-cold-query-empty branch July 17, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant