Skip to content

feat(gorilla-merger): decode-on-read for cold intchunk value chunks - #333

Merged
zzylol merged 1 commit into
mainfrom
feat/merger-coldchunk-decode
May 25, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/merger-coldchunk-decode

Conversation

@zzylol

@zzylol zzylol commented May 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the gorilla-merger the capability to decode intchunk-format cold value chunks on the read path and turn them into a standard Prometheus XOR chunk — the inverse of internal/merger/ingest.go. This is the building block for a future decode-on-read Thanos StoreAPI. The edge agent does not emit intchunk yet, so this is a tested, importable capability, not end-to-end wiring.

  • New package gorilla-merger/internal/coldchunk:
    • DecodeToSamples / DecodeChunksToSamples — wrap intchunk.DecodeChunk / DecodeChunks.
    • SamplesToXORChunk — re-encode samples as a chunkenc EncXOR chunk (the inverse of ingest.go's XOR-iterate loop).
    • DecodeToXORChunk — headline helper returning both decoded samples and the XOR chunk.
  • Unit tests encode gauge (→ INT_FOR_DELTA), counter (→ INT_FOR_DOD), and high-precision float (→ GORILLA_XOR) series so all sub-codecs are exercised, then assert (a) exact sample round-trip and (b) the produced XOR chunk iterates back to the same samples via chunkenc.

go build ./..., go vet ./..., gofmt -l, and go test ./... are all clean/green.

Build wiring (the crux)

intchunk lives in the github.com/ProjectASAP/asap-gorilla-go/intchunk subpackage, which the published asap-gorilla-go tag predates. To compile against it without cutting a new release, this PR uses the repo's existing "local replace" pattern (the same approach build_asap_otel.sh uses for asap-otel, and the data_plane crate uses for its Rust sibling path-deps):

  1. gorilla-merger/go.mod — added a local replace:

    replace github.com/ProjectASAP/asap-gorilla-go => ../../ASAPCollector/asap-gorilla-go
    

    The relative path mirrors data_plane's ../../ASAPCollector/... sibling convention: clone ASAPCollector next to ASAPQuery-backend and <repo>/gorilla-merger/../../ASAPCollector/asap-gorilla-go resolves. A directory replace adds no go.sum entries, so the committed module graph (go.sum) is unchanged. intchunk only pulls in prometheus/tsdb/chunkenc, already a merger dependency. (Note: go mod tidy was deliberately not run — this module's go.mod/go.sum intentionally retain extra Thanos-transitive deps that a tidy would prune; a pristine tidy reproduces the same large pruning, confirming it's intentional.)

  2. gorilla-merger/Dockerfile — the merger image previously fetched the published private module via a gh_token BuildKit secret. Changed to compile against the intchunk-containing source instead:

    • Accepts a BuildKit build-context asap-gorilla-go and COPY --from=asap-gorilla-go . /ASAPCollector/asap-gorilla-go (laid out as the sibling the relative replace resolves to from the /src/gorilla-merger module dir).
    • Each build RUN does go mod edit -replace github.com/ProjectASAP/asap-gorilla-go=/ASAPCollector/asap-gorilla-go so resolution is independent of WORKDIR depth.
    • The gh_token secret is now optional (used only if mounted, for any other private fetch); asap-gorilla-go itself is local source.
    • Invocation: DOCKER_BUILDKIT=1 docker build --build-context asap-gorilla-go=/path/ASAPCollector/asap-gorilla-go -t asap/gorilla-merger:dev gorilla-merger/
  3. Companion change in ASAPCollector (separate repo, not in this PR): deploy/mvp-multinode/scripts/run_demo.sh's merger build must pass the new build-context. The required edit:

        DOCKER_BUILDKIT=1 docker build --secret id=gh_token,src="${gh_token_file}" \
    +       --build-context asap-gorilla-go="${ROOT}/asap-gorilla-go" \
            -t asap/gorilla-merger:dev "${BACKEND}/gorilla-merger"

    (plus adding ${ROOT}/asap-gorilla-go to the build-inputs existence check). This must land in ASAPCollector for the cluster build to keep working with the updated Dockerfile.

Validation

  • go build/go vet/go test ./... green; gofmt -l clean.
  • Real container build verified: docker build --build-context asap-gorilla-go=<intchunk checkout> gorilla-merger/ compiled the binary against intchunk and produced the image (build step DONE, exit 0).
  • Confirmed both the workspace-mode (go.work override) and the non-workspace readonly path (directory replace, GOWORK=off) build and test green with no go.sum churn.

Test plan

  • go build ./... && go vet ./... && go test ./... in gorilla-merger/ (green)
  • gofmt -l clean
  • Container build against an intchunk-containing asap-gorilla-go checkout succeeds
  • Land the ASAPCollector run_demo.sh companion change before the next cluster deploy

🤖 Generated with Claude Code

Add internal/coldchunk, the read-side inverse of the cold ingest path: it
turns an intchunk-format value chunk (the best-of-N lossless cold codec in
asap-gorilla-go/intchunk) into samples and a standard Prometheus XOR chunk,
the building block for a future decode-on-read Thanos StoreAPI. The edge
agent does not emit intchunk yet, so this is a tested, importable capability
rather than an end-to-end wiring.

Tests encode gauge, counter, and high-precision float series so the
INT_FOR_DELTA, INT_FOR_DOD, and GORILLA_XOR sub-codecs are all exercised,
then assert exact sample round-trip and that the produced XOR chunk iterates
back to the same samples.

intchunk lives in a subpackage the published asap-gorilla-go tag predates, so
resolve it WITHOUT a new release: add a local replace pointing asap-gorilla-go
at the in-repo monorepo checkout (relative ../../ASAPCollector/asap-gorilla-go,
mirroring the data_plane crate's sibling path-deps). The container build
supplies that checkout as a BuildKit build-context and rewrites the replace to
the in-image path, so the merger image always compiles against the
intchunk-containing source. A directory replace adds no go.sum entries, so the
committed module graph is unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 403c8d4 into main May 25, 2026
@zzylol
zzylol deleted the feat/merger-coldchunk-decode branch July 17, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant