fix(gorillas3): TSDB appender out-of-bounds with rotating-cardinality metrics

[zzylol]

Bug

Agent processor crashed 60s after start with:

2026-05-08T22:10:14.679Z error gorillas3processor/processor.go:325 gorillas3: tsdb block build failed
  error="tsdb appender.Append: out of bounds"
  at flushTSDB → flushWindow → Start.func1

Reproducible 100% of the time after PR #338's high-cardinality rotating metrics (unique_users_per_min, top_endpoint_qps) started flowing. Blocks the entire MVP demo.

Root cause (hypothesis (a) — timestamp ordering)

Prometheus TSDB's head.Appender() snapshots minValidTime = appendableMinValidTime() = max(MaxTime - chunkRange/2, headMinValid) at the time the appender is created (or, for an empty Head, on first Append). For a fresh BlockWriter Head this means minValidTime = firstAppendedSampleTs - chunkRange/2 — fixed for the appender's lifetime.

With a 60s tsdb_block_duration and Go's randomised map iteration over the window, if the first-visited series held samples in e.g. the [40..55s] slice of the window, the floor locked at 10s = 40 - 30. Any subsequent series with samples at t<10s then tripped storage.ErrOutOfBounds.

PR #338's unique_users_per_min rotates user_ids over the window — different user_id series have NON-OVERLAPPING per-series timestamp ranges by design — which made cross-series-ordering OOB fire on every flush.

Fix

opentelemetry-collector-contrib-patch/processor/gorillas3processor/

1. tsdb_block_writer.go appendWindow: flatten every (labelset, ts, v) tuple across all series, sort globally by timestamp ascending, then drive app.Append. Guarantees the first appended sample is the smallest ts in the window, so minValidTime = min(window) - 30s sits below every other sample. (The per-series ascending sort is preserved as a stable secondary order.)
2. Defensive: errors.Is(err, storage.ErrOutOfBounds) from Append is now NON-fatal — the sample is skipped, a counter is bumped, and the appender continues. The artifact surfaces a NumOOBDropped count.
3. selfmonitor.go: new gorillas3_tsdb_oob_samples_dropped_total counter so operators see drift.
4. processor.go flushTSDB: log a Warn (not Error) per-flush when drops occur; bump the counter.

Test coverage

tsdb_block_writer_test.go:

• TestTSDBBlockBuilder_RotatingCardinalityNoOOB — 50 trials of randomised map iteration over 8 series with non-overlapping 7-second slices in a 60s window. Fails on the unfixed code with the exact error from production; passes on the fix every iteration.
• TestFlushTSDB_OOBDoesNotCrashProcessor — drives the same scenario through the full ConsumeMetrics → flushWindow → flushTSDB → PutTSDBBlock path; round-trips the block via tsdb.OpenBlock and verifies all 56 samples land.
• TestTSDBBlockBuilder_WideTimestampSpanNoOOB — 5-minute span between earliest and latest samples in a single window builds cleanly.

Before/after evidence

Reproducer on baseline (unfixed) main:

=== RUN   TestIssue46Repro
    issue46_repro_test.go:50:
        Error: tsdb appender.Append: out of bounds
        Test:  TestIssue46Repro
        Messages: trial 0
--- FAIL: TestIssue46Repro (0.01s)

After fix:

=== RUN   TestTSDBBlockBuilder_RotatingCardinalityNoOOB
--- PASS: TestTSDBBlockBuilder_RotatingCardinalityNoOOB (2.90s)
=== RUN   TestFlushTSDB_OOBDoesNotCrashProcessor
--- PASS: TestFlushTSDB_OOBDoesNotCrashProcessor (0.06s)
PASS
ok  github.com/open-telemetry/opentelemetry-collector-contrib/processor/gorillas3processor   6.275s

Closes #46 (gorillas3 TSDB OOB blocker).

Test plan

• Unit tests pass (full package)
• Reproducer fails on baseline, passes after fix
• Live MVP soak — deferred; worktree's submodule chain isn't populated, and a parallel agent is reducing fake-exporter cardinality. The unit-test reproducer captures the exact failure path.

🤖 Generated with Claude Code