gorilla-merger: Thanos-Receive-style fragment->TSDB merger (write path + StoreAPI)

.design_docs/GORILLA_MERGER_DESIGN.md

@@ -0,0 +1,147 @@
+# Gorilla Merger — design (Go, co-located; Thanos-Receive pattern)
+
+Status: draft for review. Supersedes the GORILLA1 compactor direction AND the
+earlier Model-A (finalizer-on-cut, accept-the-lag) variant.
+
+## Goal
+
+Cold/raw tier: agents emit compact, Gorilla-XOR-compressed fragments; the
+backend merges ~1min agent chunks into large **2h Prometheus TSDB blocks** with
+**one S3 PUT per block** (minimize S3 insert cost), serves the **pending <2h
+window** for fallback exact queries, and lets queries **union** pending + S3.
+Small CPU/mem/disk where possible.
+
+## Decisions
+
+- **Final S3 format = Prometheus TSDB blocks** (Thanos-queriable). Only proven
+  TSDB writer is Go's `prometheus/tsdb`, so the merger is **Go, co-located**
+  with the backend (Option #1; Rust/cgo rejected — no perf gain, hybrid-build
+  cost).
+- **Edge→merger wire = XOR-chunk fragments** (~1.3 B/sample) via a **shared
+  binary codec in `github.com/ProjectASAP/asap-gorilla-go`** (both the edge
+  encoder and merger decoder import it — single source of truth, no JSON/base64).
+- **2h merge window, one PUT per block.** Thanos **Compactor is optional**
+  (blocks are already 2h). Agents do NOT write blocks or PUT to S3.
+- **Merger = Thanos-Receive pattern** (this is the load-bearing decision, set by
+  the requirement to query the pending window): an embedded `tsdb.DB` (2h block
+  range + WAL) fed by a fragment HTTP frontend, exposed as a **Thanos StoreAPI**
+  via `thanos store.TSDBStore`, with a `thanos shipper` uploading cut blocks to
+  S3. "Thanos Receive with an XOR-fragment frontend instead of remote-write."
+- **Union via thanos-query.** thanos-query fans out to the merger's StoreAPI
+  (<2h pending) + thanos-store-gateway (≥2h S3). The backend's existing
+  `ThanosQueryEngine` (HTTP-forward to thanos-query) is UNCHANGED.
+- **GORILLA1 is superseded** and slated for deletion (the codebase's own "Phase
+  δ" legacy-leg removal; task #30).
+
+## Topology
+
+```
+EDGE (OTel collector, asap_edge cold)          BACKEND merger (NEW Go binary, co-located)
+─────────────────────────────────────          ──────────────────────────────────────────
+asap_edge shared decode + key (per dp)
+  └─ cold: StreamingFragmentEncoder
+       raw -> XOR chunk (chunkenc)
+       watermark-bounded, no index/S3
+  └─ ship via shared binary codec  ──HTTP──▶    POST /ingest/gorilla
+       (gorilla.EncodeFragmentBatch, gzip)        └─ gorilla.DecodeFragmentBatch
+                                                   └─ tsdb.DB.Appender().Append(labels,t,v)
+                                                 tsdb.DB: 2h head + WAL  (the pending window)
+                                                 store.TSDBStore ── Thanos StoreAPI (gRPC) ◀─┐
+                                                 shipper: cut 2h block -> ONE PUT -> S3      │
+                                                                                  │          │
+thanos-query  fan-out + union ───────────────────────────────────────────────────│──────────┘
+   ├─ merger StoreAPI         (recent, <2h pending)                               │
+   └─ thanos-store-gateway ── S3 TSDB blocks (≥2h) ◀──────────────────────────────┘
+
+ASAP fallback PromQL ─▶ ThanosQueryEngine (forward.rs, UNCHANGED) ─▶ thanos-query ─▶ union
+```
+
+## Wire contract (shared codec, task #31)
+
+Add to `asap-gorilla-go` (imported by both repos):
+`EncodeFragmentBatch([]Fragment) []byte` / `DecodeFragmentBatch([]byte)
+([]Fragment, error)`. Binary frame (HTTP POST body, `Content-Encoding: gzip`):
+
+```
+magic "ASAPFRG1" (8B) | version u8 | fragment_count uvarint
+repeat:
+  metric_name      : uvarint len + bytes
+  label_count      : uvarint
+  repeat: name (uvarint len+bytes), value (uvarint len+bytes)   # sorted by name
+  min_time_ms      : varint (zigzag)
+  max_time_ms      : varint (zigzag)
+  sample_count     : uvarint
+  encoding         : u8 (0 = XOR / chunkenc.EncXOR)
+  chunk_len        : uvarint
+  chunk_bytes      : [chunk_len]   # the raw chunkenc XOR chunk
+```
+
+No base64, no JSON (drops +33% + reflection). Replaces the OTLP-attribute
+`MarshalFragment` smuggling.
+
+## Edge (Track 1, task #29)
+
+- `asap_edge` cold tier swaps `StreamingTSDBBlockBuilder` → `gorilla.
+  StreamingFragmentEncoder` (raw → `chunkenc` XOR chunks on a watermark; no
+  index, no S3). Reuse the fused shared decode + `gorilla.SeriesKey`.
+- Replace `blockShipper` (tars a TSDB block) with a **fragment shipper**:
+  `gorilla.EncodeFragmentBatch` → gzip → `POST cold.ship_endpoint`
+  (`/ingest/gorilla`). Batched per flush tick; ret/backoff as today.
+- Edge stays cheap: bounded OOO buffer, no block build, no S3.
+
+## Merger (Track 2, task #26) — Thanos-Receive style
+
+New Go module in the backend repo (own `go.mod`, imports `asap-gorilla-go` +
+`prometheus/prometheus/tsdb` + `thanos-io/thanos`). Proposed `gorilla-merger/`
+with `cmd/gorilla-merger/main.go`.
+
+### Ingest
+- `POST /ingest/gorilla`: gunzip → `gorilla.DecodeFragmentBatch` → for each
+  fragment, `chunkenc.FromData(EncXOR, data)` → iterate samples → `appender.
+  Append(labels, t, v)`. 200 after the appender commits (WAL durable).
+- Cross-agent fan-in: same labels (incl. agent/source external label) → same
+  series in the DB. Distinct agents carry a distinguishing external label.
+
+### Open-window store + StoreAPI (the union mechanism)
+- `tsdb.Open(dir, …)` with `MinBlockDuration = MaxBlockDuration = 2h` → the head
+  holds the pending window; WAL = durability + bounded mem.
+- `store.NewTSDBStore(db, externalLabels, component)` → serve the **Thanos
+  StoreAPI** over gRPC. Register this endpoint in **thanos-query**'s store list
+  (task #32) so it fans in alongside store-gateway.
+
+### Cut → ship (one PUT/block)
+- Head auto-compacts at the 2h boundary → block on local disk.
+- `shipper.New(…, bucket, …)` watches the dir, uploads each new block to S3
+  (one PUT set per block: chunks + index + meta.json) with the **Thanos
+  `thanos{}` meta** (shipper writes it). Local retention short — drop blocks
+  once shipped + store-gateway has them.
+
+### Cost levers
+- CPU: decode→append (XOR→samples; centralized). No agent-side index build.
+- Mem: 2h head (in-mem chunks) + WAL — conservative vs Receive's 2h default.
+- Disk: WAL + the not-yet-shipped block(s); bounded by ship cadence + retention.
+- S3: ONE PUT per 2h block (the explicit goal).
+
+## Read path — UNCHANGED in the backend
+
+`ThanosQueryEngine` (`data_plane/.../thanos_query_engine/forward.rs`) HTTP-
+forwards archive PromQL to thanos-query (`ASAP_THANOS_QUERY_URL`, default
+`http://thanos-query:10903`). thanos-query unions merger-StoreAPI + store-
+gateway. The legacy `GorillaS3Store` (GORILLA1) leg is deleted in Phase δ (#30).
+Recency for warm/sketch queries is still served by the warm tier; the freshness
+probe cache is unaffected.
+
+## Cleanup (Track 3)
+- Rewire `main.rs` archive selection to Path A2 (set `ASAP_THANOS_QUERY_URL`);
+  delete legacy `GorillaS3Store` leg (#30).
+- Delete GORILLA1: Go encoder (`gorilla.go` Build*/EncodeSeriesBody + bit
+  encoders), Rust store/decoder, abandoned `compactor.rs`, stale strings (#30).
+- Retire `gateway_fragment` OTel role (#27).
+
+## Deploy additions
+- thanos-query store list += merger StoreAPI endpoint (#32).
+- merger S3 bucket = the bucket thanos-store-gateway watches.
+
+## Non-goals
+- No change to warm tier (sum + sketches) output.
+- No downsampling in the merger (Thanos Compactor owns it, and is optional).

gorilla-merger/.gitignore

@@ -0,0 +1,4 @@
+# Local tsdb data dir (WAL + unshipped blocks) and built binary.
+/data/
+/gorilla-merger
+*.test

gorilla-merger/Dockerfile

@@ -0,0 +1,69 @@
+# syntax=docker/dockerfile:1.7
+#
+# Multi-stage build for the gorilla-merger (Thanos-Receive-style component).
+#
+# ## Why this needs a build secret
+#
+# gorilla-merger imports the PRIVATE Go module
+# `github.com/ProjectASAP/asap-gorilla-go` (the shared edge<->merger
+# ASAPFRG1 wire codec). A naive `go build` inside Docker/CI cannot fetch it:
+# the module proxy + git fetch get a 404/auth prompt for the private repo.
+#
+# We solve this with a BuildKit *secret* mount carrying a GitHub token. The
+# secret is mounted ONLY for the duration of the build RUN that needs it and
+# is NOT baked into any image layer (unlike a build-arg or COPY of a token
+# file). Inside that RUN we set a transient `url.insteadOf` git rewrite so
+# `go` fetches the private module over HTTPS with the token. The git config
+# lives only in the container build layer, never on the host.
+#
+# `GOPRIVATE=github.com/ProjectASAP/*` keeps the fetch off the public proxy
+# and checksum DB; `GIT_TERMINAL_PROMPT=0` makes a missing/incorrect token
+# fail fast instead of hanging on an interactive credential prompt.
+#
+# ## Build invocation
+#
+# Write a GitHub token (a PAT or `gh auth token`) to a file, then:
+#
+#   gh auth token > /tmp/gh_token            # or: echo "$GITHUB_TOKEN" > /tmp/gh_token
+#   DOCKER_BUILDKIT=1 docker build \
+#       --secret id=gh_token,src=/tmp/gh_token \
+#       -t asap/gorilla-merger:dev \
+#       gorilla-merger/
+#   rm -f /tmp/gh_token
+#
+# The build context is the `gorilla-merger/` module directory (this file's
+# directory). The same GOPRIVATE + token requirement applies to
+# ASAPQuery-backend CI before PR #310 can be merged (the CI runner must
+# expose a `gh_token` secret / configure `url.insteadOf` the same way).
+
+FROM golang:1.25-bookworm AS build
+WORKDIR /src
+
+# git is needed for the private-module fetch (insteadOf rewrite below).
+RUN apt-get update && apt-get install -y --no-install-recommends git ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+# Prime the module graph first so dependency downloads cache independently
+# of source edits. go.sum is committed, so the public deps resolve normally;
+# only the private module needs the token.
+COPY go.mod go.sum ./
+RUN --mount=type=secret,id=gh_token \
+    GOPRIVATE=github.com/ProjectASAP/* \
+    GONOSUMCHECK=github.com/ProjectASAP/* \
+    GOFLAGS=-mod=mod \
+    GIT_TERMINAL_PROMPT=0 \
+    sh -c 'git config --global url."https://x-access-token:$(cat /run/secrets/gh_token)@github.com/".insteadOf "/" && go mod download'
+
+# Now copy the rest of the module and build.
+COPY . .
+RUN --mount=type=secret,id=gh_token \
+    GOPRIVATE=github.com/ProjectASAP/* \
+    GONOSUMCHECK=github.com/ProjectASAP/* \
+    GIT_TERMINAL_PROMPT=0 \
+    sh -c 'git config --global url."https://x-access-token:$(cat /run/secrets/gh_token)@github.com/".insteadOf "/" && CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/gorilla-merger ./cmd/gorilla-merger'
+
+FROM gcr.io/distroless/static-debian12:nonroot
+COPY --from=build /out/gorilla-merger /usr/local/bin/gorilla-merger
+# HTTP ingest (/ingest/gorilla, /metrics) + Thanos StoreAPI gRPC.
+EXPOSE 10908 10907
+ENTRYPOINT ["/usr/local/bin/gorilla-merger"]

gorilla-merger/README.md

@@ -0,0 +1,69 @@
+# gorilla-merger
+
+A Thanos-Receive-style component for ASAP edge agents. It:
+
+1. **Ingests** Gorilla XOR-chunk fragments (the `asap-gorilla-go` `ASAPFRG1`
+   wire codec) over HTTP `POST /ingest/gorilla` (gzip-aware).
+2. **Appends** the decoded samples to an embedded Prometheus `tsdb.DB`
+   (2h block range + WAL).
+3. **Serves** a Thanos **StoreAPI** (gRPC) over the open (`<2h` pending)
+   window so `thanos-query` can union recent data with the `>=2h` S3 blocks
+   that `thanos-store-gateway` serves.
+4. **Ships** completed 2h blocks to object storage via the Thanos shipper
+   (one PUT set per block), into the same bucket the store-gateway watches.
+
+## Ports / flags
+
+| Flag | Env | Default | Purpose |
+|------|-----|---------|---------|
+| `-http-address` | `MERGER_HTTP_ADDRESS` | `:10908` | `/ingest/gorilla`, `/metrics`, `/-/healthy`, `/-/ready` |
+| `-grpc-address` | `MERGER_GRPC_ADDRESS` | `:10907` | Thanos StoreAPI (the query surface `thanos-query --store=` points at) |
+| `-tsdb.path` | `MERGER_TSDB_PATH` | `./data` | embedded tsdb dir (WAL + unshipped blocks) |
+| `-objstore.config-file` | `MERGER_OBJSTORE_CONFIG_FILE` | _empty_ | Thanos objstore YAML; empty disables the shipper |
+| `-external-labels` | `MERGER_EXTERNAL_LABELS` | _empty_ | `k=v,k=v` applied to every series + uploaded block; distinct mergers MUST carry a distinguishing label |
+| `-shipper.interval` | `MERGER_SHIPPER_INTERVAL` | `1m` | block-scan / upload cadence |
+| `-tsdb.retention` | `MERGER_TSDB_RETENTION` | `6h` | local on-disk retention (blocks live in S3 once shipped) |
+
+## Building the container
+
+The merger imports the **private** Go module
+`github.com/ProjectASAP/asap-gorilla-go`, so a naive `go build` in
+Docker/CI cannot fetch it (404/auth-prompt on the private repo). The
+`Dockerfile` solves this with a **BuildKit secret** carrying a GitHub
+token — the token is mounted only for the build `RUN`s that need it and is
+never baked into an image layer (unlike a build-arg or a `COPY`ed token
+file). The git `url.insteadOf` rewrite happens *inside* the container
+build, never on the host.
+
+```sh
+# Write a GitHub token to a file. With a modern gh:        gh auth token > /tmp/gh_token
+# With gh < 2.x (no `gh auth token` subcommand) read it from the gh config:
+python3 -c "import yaml; d=yaml.safe_load(open('$HOME/.config/gh/hosts.yml')); print(d['github.com'].get('oauth_token') or d['github.com'].get('token'), end='')" > /tmp/gh_token
+# ...or just:        echo "$GITHUB_TOKEN" > /tmp/gh_token
+
+DOCKER_BUILDKIT=1 docker build \
+    --secret id=gh_token,src=/tmp/gh_token \
+    -t asap/gorilla-merger:dev \
+    .                                    # build context = this directory
+
+rm -f /tmp/gh_token
+```
+
+The token needs `repo` read scope on `github.com/ProjectASAP/asap-gorilla-go`.
+
+### CI note (PR #310)
+
+The **same** `GOPRIVATE=github.com/ProjectASAP/*` + token requirement
+applies to ASAPQuery-backend CI before PR #310 can merge: the CI runner
+must expose a `gh_token` secret (or set `url.insteadOf` with a token) so
+`go build` / `go test` of `gorilla-merger/` can fetch the private module.
+
+## Local dev (no container)
+
+`go vet` / `go build` on a host that already has the module in its
+`GOMODCACHE` (or with `git` configured for the private repo):
+
+```sh
+GOPRIVATE=github.com/ProjectASAP/* go vet ./...
+GOPRIVATE=github.com/ProjectASAP/* go build ./cmd/gorilla-merger
+```