fix(gorillas3processor): single critical section for ensureRoleState + consume/flush (TOCTOU race)

[zzylol]

Summary

ConsumeMetrics (line 114) and flushWindow (line 298) each called
ensureRoleState() which took p.mu, optionally created
p.rawBuilder / p.fragmentEncoder / p.fragmentFinalizer, then
released p.mu — and then the callers re-acquired p.mu to do their
real work. Between the two acquisitions, another goroutine could
mutate the role state.

Concretely for the default (gateway-raw) role:

1. ConsumeMetrics → ensureRoleState() takes p.mu, lazily creates
   p.rawBuilder, releases p.mu.
2. Timer goroutine flushWindow runs, takes p.mu, calls
   flushGatewayRawLocked which at line 337 does
   p.rawBuilder = nil (the next sample is expected to lazily
   recreate it), releases p.mu.
3. ConsumeMetrics re-acquires p.mu and calls
   consumeGatewayRawLocked → ingestRawMetricLocked →
   p.rawBuilder.AddSample → SIGSEGV (rawBuilder is now nil).

Observed agent panic in the multinode harness:

panic: runtime error: invalid memory address or nil pointer dereference
asap-gorilla-go.(*StreamingTSDBBlockBuilder).AddSample(0x0, ...)
gorillas3processor/processor.go:233

The agent crash-loops every ~10s, which silently breaks the DDSketch
flush path. PR #394's multinode validation masked this because
sum by (zone) (http_requests_total) travels via raw_passthrough
and is aggregated at the backend; quantile_over_time(_latency_ms)
queries fail because the DDSketch state never makes it across.

Fix (Option 2)

• Rename ensureRoleState → ensureRoleStateLocked and remove its
  internal p.mu.Lock()/Unlock(). Caller's responsibility now.
• ConsumeMetrics: take p.mu, call ensureRoleStateLocked, then
  do consume work — all inside the SAME critical section.
• flushWindow: take p.mu, call ensureRoleStateLocked, then
  dispatch to flush{Agent,GatewayFragment,GatewayRaw}Locked.
• Start: take/release p.mu around the init call (the
  background flush goroutine isn't running yet, but the lock is
  cheap and keeps the contract uniform).

The same fix protects the flushAgentLocked /
flushGatewayFragmentLocked paths that also nil out
p.fragmentEncoder / p.fragmentFinalizer after a successful flush.

Drive-by harness fix

deploy/mvp-multinode/scripts/run_demo.sh invokes
thanos-query with no --grpc-address, which defaults to
:10901 — that collides with thanos-store-gateway (also :10901)
under --network host on node2. :10902 also collides (store-gateway
HTTP). Pin query's gRPC listener to :10905 so all three thanos
containers coexist. Without this, the 5-query validation can't run
at all — asap-backend's thanos_query engine returns
connection-refused. Only --grpc-address is added; nothing else
moves.

Validation (multinode)

go test -race ./... clean (3.5s).

Crash-loop / nil-deref counters after 5 min uptime under load:

signal	node0 agent-a	node3 agent-b	node1 gateway
panic|SIGSEGV count	0	0	0
AddSample (nil deref site)	0	0	n/a
status	Up 5 minutes	Up 5 minutes	Up 5 minutes

5-query gate (all status:success, all data_source: asap_query):

=== quantile_over_time(0.99, http_requests_total_latency_ms[5m]) ===
result[0]: zone=z3 value=104.6   result[1]: zone=z2 value=122.75
result[2]: zone=z1 value=89.13   result[3]: zone=z3 value=115.6  ... (many series)

=== quantile_over_time(0.5,  http_requests_total_latency_ms[5m]) ===
result[0]: zone=z3 value=17.99   result[1]: zone=z2 value=...    ... (many series)

=== sum by (zone) (http_requests_total) ===
z0=305962242  z1=305966329  z2=305964653  z3=305965054

=== sum by (zone) (rate(http_requests_total[5m])) ===
z0=2.257e6  z1=2.258e6  z2=2.258e6  z3=2.258e6

=== topk(5, sum by (zone) (rate(http_requests_total[5m]))) ===
z1=2.258e6  z3=2.258e6  z2=2.258e6  z0=2.257e6

The two quantile queries are the regression check — they failed
pre-fix because the agent's DDSketch path crash-looped and never
shipped state. They pass post-fix. The three sum/topk queries
continue working, so PR #394's behavior is preserved.

Test plan

• go build ./... clean
• go test ./... clean
• go test -race ./... clean
• Image rebuilt + synced to node0/node1/node3 (same Created timestamp on all three)
• Multinode up asap — no crash loop on agents or gateway
• All 5 queries return status:success with data_source: asap_query
• Quantile queries (regression check) return real DDSketch-backed values
• sum-by-zone / rate / topk continue working (no fix(deploy): wire controller into multinode harness + replay typo + wave queries + stale bucket key #394 regression)

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com