feat: coldpart Part format in asap-gorilla-go + agent intchunk cold-part ship path - #439
Merged
Merged
Conversation
…-part ship path Move the cold "Part" format (intchunk-coded read/write library) into asap-gorilla-go/coldpart so both the edge agent (producer) and the backend gorilla-merger (consumer) can import it; it previously lived in the merger's internal package, which the agent cannot import. Add an opt-in intchunk cold-part path to the asap_edge processor, parallel to the default gorilla-XOR fragment ship path (unchanged when the flag is unset): when cold.format=intchunk the same per-window drained samples are re-encoded as a lossless coldpart.Part with absolute-ms block bounds and POSTed to the merger's /ingest/coldpart (cold.coldpart_endpoint). Series labels are built via the new exported gorilla.FragmentLabels so either format archives a series under one identity; gorilla.DecodeFragmentSamples recovers raw samples from the shared fragments. This is the first of a two-repo change. A follow-up ASAPQuery-backend PR re-imports coldpart from asap-gorilla-go and deletes its internal copy; merge this PR first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First of a two-PR change (merge this one first; the ASAPQuery-backend re-import PR follows).
coldpartto the shared lib. Addsasap-gorilla-go/coldpart(full read+write code + tests), next tointchunk, so BOTH the edge agent (producer) and the backend gorilla-merger (consumer) can import the cold "Part" format. The merger'sinternal/coldpartis deleted in the follow-up backend PR.asap_edgeprocessor. Whencold.format: intchunk(default is the existing gorilla-XORfragmentpath, unchanged), the SAME per-window drained samples are re-encoded as a losslesscoldpart.Partwith absolute-ms block bounds and POSTed to the merger'sPOST /ingest/coldpart(cold.coldpart_endpoint). The merger derives the content-addressed key itself; the agent sends only the bytes.gorilla.FragmentLabels(single source of truth for fragment→labels, also used by the fragment→TSDB finalizer) guarantees a series is queried under one identity regardless of cold format;gorilla.DecodeFragmentSamplesrecovers raw samples from the same drained XOR fragments.The flag is wired through
ColdConfig(the same config path that selects the current cold ship behavior); an operator typically sets it from an env var, e.g.cold.format: ${env:ASAP_COLD_FORMAT}withASAP_COLD_FORMAT=intchunk.Test plan
asap-gorilla-go:go build ./...,go test ./...(incl. movedcoldparttests),go vet,gofmtall clean.asapedgeprocessor: builds + fullgo test ./...green;go vet/gofmtclean.coldpart.OpenPart+Series()round-trips samples bit-exactly with absolute-ms block bounds; POST to an in-process/ingest/coldpart-style handler validates viaOpenPart; fullConsumeMetrics→flushAllexercises the opt-in path end to end.cold.formatunset the processor keeps the fragment path and wires no cold-part shipper (no behavior change).🤖 Generated with Claude Code