feat(control_plane): emit cold-archive intchunk format to the edge agent - #340
Merged
Merged
Conversation
Add a cold-archive format knob so the control plane can tell the edge agent to ship the lossless intchunk cold-part format instead of the default gorilla-XOR fragments. Adds `cold_format` (enum: fragment default | intchunk) and `cold_coldpart_endpoint` to `EdgeStageConfig`. The fused `asap_edge` emit now writes `format: intchunk` + `coldpart_endpoint:` into the agent `cold:` block only when the format is intchunk; the default fragment path emits neither key, leaving the cold block byte-identical to today. Operators opt in per deployment via the `ASAP_COLD_FORMAT=intchunk` env knob (mirrors how `ASAP_CLUSTER` drives the cold external-label default), read in the shared bootstrap/replan stitch. The coldpart endpoint is derived from the fragment ship endpoint (same merger host:port, `/ingest/coldpart` path) unless overridden. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
asapedgeprocessoralready supports both (cold.format: fragment→/ingest/gorilla,cold.format: intchunk→cold.coldpart_endpoint/ingest/coldpart); the control plane previously had no way to emitformat:/coldpart_endpoint:, so the agent always got fragments.EdgeStageConfigfields:cold_format(enumfragmentdefault |intchunk) andcold_coldpart_endpoint: Option<String>(inphysical/colored_dag/emitter.rs), with named defaultsdefault_cold_coldpart_endpoint()+ acoldpart_endpoint_from_ship()deriver. Both#[serde(skip_serializing_if ...)]so an unset format is byte-identical on the wire.asap_edgecold-block builder (emit/stage_config.rs::emit_edge_yaml_asap_edge) now writesformat: intchunk+coldpart_endpoint:only whencold_format == Intchunk. The default fragment path emits neither key.ASAP_COLD_FORMAT=intchunkenv knob, read in the shared bootstrap/replan stitch (emit/mod.rs::apply_cold_format_from_env, invoked fromextend_edge_with_demo_plumbing) — mirrors howASAP_CLUSTERdrives the cold external-label default. The coldpart endpoint is derived from the fragment ship endpoint (same merger host:port,/ingest/coldpart) unlessASAP_COLD_COLDPART_ENDPOINToverrides it.fragment— no behavior change when unset. Data plane / warm path untouched.Test plan
cargo build -p control_planegreen (no new warnings).cargo test -p control_planegreen: 779 lib + 28 bin pass.cold_format_default_fragment_emits_no_format_keys(fragment → noformat:/coldpart_endpoint:key),cold_format_intchunk_emits_format_and_derived_coldpart_endpoint(intchunk → both keys, endpoint derived),cold_format_intchunk_honours_explicit_coldpart_endpoint, andcold_format_env_knob_opts_into_intchunk_and_derives_endpoint(theASAP_COLD_FORMATSET path).Generated with Claude Code