feat(gorilla-merger): decode-on-read for cold intchunk value chunks - #333
Merged
Merged
Conversation
Add internal/coldchunk, the read-side inverse of the cold ingest path: it turns an intchunk-format value chunk (the best-of-N lossless cold codec in asap-gorilla-go/intchunk) into samples and a standard Prometheus XOR chunk, the building block for a future decode-on-read Thanos StoreAPI. The edge agent does not emit intchunk yet, so this is a tested, importable capability rather than an end-to-end wiring. Tests encode gauge, counter, and high-precision float series so the INT_FOR_DELTA, INT_FOR_DOD, and GORILLA_XOR sub-codecs are all exercised, then assert exact sample round-trip and that the produced XOR chunk iterates back to the same samples. intchunk lives in a subpackage the published asap-gorilla-go tag predates, so resolve it WITHOUT a new release: add a local replace pointing asap-gorilla-go at the in-repo monorepo checkout (relative ../../ASAPCollector/asap-gorilla-go, mirroring the data_plane crate's sibling path-deps). The container build supplies that checkout as a BuildKit build-context and rewrites the replace to the in-image path, so the merger image always compiles against the intchunk-containing source. A directory replace adds no go.sum entries, so the committed module graph is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the gorilla-merger the capability to decode intchunk-format cold value chunks on the read path and turn them into a standard Prometheus XOR chunk — the inverse of
internal/merger/ingest.go. This is the building block for a future decode-on-read Thanos StoreAPI. The edge agent does not emit intchunk yet, so this is a tested, importable capability, not end-to-end wiring.gorilla-merger/internal/coldchunk:DecodeToSamples/DecodeChunksToSamples— wrapintchunk.DecodeChunk/DecodeChunks.SamplesToXORChunk— re-encode samples as achunkencEncXOR chunk (the inverse of ingest.go's XOR-iterate loop).DecodeToXORChunk— headline helper returning both decoded samples and the XOR chunk.INT_FOR_DELTA), counter (→INT_FOR_DOD), and high-precision float (→GORILLA_XOR) series so all sub-codecs are exercised, then assert (a) exact sample round-trip and (b) the produced XOR chunk iterates back to the same samples viachunkenc.go build ./...,go vet ./...,gofmt -l, andgo test ./...are all clean/green.Build wiring (the crux)
intchunklives in thegithub.com/ProjectASAP/asap-gorilla-go/intchunksubpackage, which the published asap-gorilla-go tag predates. To compile against it without cutting a new release, this PR uses the repo's existing "local replace" pattern (the same approachbuild_asap_otel.shuses for asap-otel, and the data_plane crate uses for its Rust sibling path-deps):gorilla-merger/go.mod— added a localreplace:The relative path mirrors data_plane's
../../ASAPCollector/...sibling convention: clone ASAPCollector next to ASAPQuery-backend and<repo>/gorilla-merger/../../ASAPCollector/asap-gorilla-goresolves. A directory replace adds no go.sum entries, so the committed module graph (go.sum) is unchanged. intchunk only pulls inprometheus/tsdb/chunkenc, already a merger dependency. (Note:go mod tidywas deliberately not run — this module's go.mod/go.sum intentionally retain extra Thanos-transitive deps that a tidy would prune; a pristine tidy reproduces the same large pruning, confirming it's intentional.)gorilla-merger/Dockerfile— the merger image previously fetched the published private module via agh_tokenBuildKit secret. Changed to compile against the intchunk-containing source instead:asap-gorilla-goandCOPY --from=asap-gorilla-go . /ASAPCollector/asap-gorilla-go(laid out as the sibling the relative replace resolves to from the/src/gorilla-mergermodule dir).RUNdoesgo mod edit -replace github.com/ProjectASAP/asap-gorilla-go=/ASAPCollector/asap-gorilla-goso resolution is independent of WORKDIR depth.gh_tokensecret is now optional (used only if mounted, for any other private fetch); asap-gorilla-go itself is local source.DOCKER_BUILDKIT=1 docker build --build-context asap-gorilla-go=/path/ASAPCollector/asap-gorilla-go -t asap/gorilla-merger:dev gorilla-merger/Companion change in ASAPCollector (separate repo, not in this PR):
deploy/mvp-multinode/scripts/run_demo.sh's merger build must pass the new build-context. The required edit:DOCKER_BUILDKIT=1 docker build --secret id=gh_token,src="${gh_token_file}" \ + --build-context asap-gorilla-go="${ROOT}/asap-gorilla-go" \ -t asap/gorilla-merger:dev "${BACKEND}/gorilla-merger"(plus adding
${ROOT}/asap-gorilla-goto the build-inputs existence check). This must land in ASAPCollector for the cluster build to keep working with the updated Dockerfile.Validation
go build/go vet/go test ./...green;gofmt -lclean.docker build --build-context asap-gorilla-go=<intchunk checkout> gorilla-merger/compiled the binary against intchunk and produced the image (build step DONE, exit 0).Test plan
go build ./... && go vet ./... && go test ./...ingorilla-merger/(green)gofmt -lcleanrun_demo.shcompanion change before the next cluster deploy🤖 Generated with Claude Code