Skip to content

feat(control_plane): emit cold-archive intchunk format to the edge agent - #340

Merged
zzylol merged 1 commit into
mainfrom
feat/cold-archive-format-emit
May 25, 2026
Merged

zzylol merged 1 commit into
mainfrom
feat/cold-archive-format-emit

Conversation

@zzylol

@zzylol zzylol commented May 25, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add a cold-archive format knob so the control plane can tell the edge agent to ship the lossless intchunk cold-part format instead of the default gorilla-XOR fragments. The agent's asapedgeprocessor already supports both (cold.format: fragment/ingest/gorilla, cold.format: intchunkcold.coldpart_endpoint /ingest/coldpart); the control plane previously had no way to emit format:/coldpart_endpoint:, so the agent always got fragments.
  • New EdgeStageConfig fields: cold_format (enum fragment default | intchunk) and cold_coldpart_endpoint: Option<String> (in physical/colored_dag/emitter.rs), with named defaults default_cold_coldpart_endpoint() + a coldpart_endpoint_from_ship() deriver. Both #[serde(skip_serializing_if ...)] so an unset format is byte-identical on the wire.
  • The fused asap_edge cold-block builder (emit/stage_config.rs::emit_edge_yaml_asap_edge) now writes format: intchunk + coldpart_endpoint: only when cold_format == Intchunk. The default fragment path emits neither key.
  • Operator SET path: ASAP_COLD_FORMAT=intchunk env knob, read in the shared bootstrap/replan stitch (emit/mod.rs::apply_cold_format_from_env, invoked from extend_edge_with_demo_plumbing) — mirrors how ASAP_CLUSTER drives the cold external-label default. The coldpart endpoint is derived from the fragment ship endpoint (same merger host:port, /ingest/coldpart) unless ASAP_COLD_COLDPART_ENDPOINT overrides it.
  • Default stays fragment — no behavior change when unset. Data plane / warm path untouched.

Test plan

  • cargo build -p control_plane green (no new warnings).
  • cargo test -p control_plane green: 779 lib + 28 bin pass.
  • New emit tests: cold_format_default_fragment_emits_no_format_keys (fragment → no format:/coldpart_endpoint: key), cold_format_intchunk_emits_format_and_derived_coldpart_endpoint (intchunk → both keys, endpoint derived), cold_format_intchunk_honours_explicit_coldpart_endpoint, and cold_format_env_knob_opts_into_intchunk_and_derives_endpoint (the ASAP_COLD_FORMAT SET path).

Generated with Claude Code

Add a cold-archive format knob so the control plane can tell the edge
agent to ship the lossless intchunk cold-part format instead of the
default gorilla-XOR fragments.

Adds `cold_format` (enum: fragment default | intchunk) and
`cold_coldpart_endpoint` to `EdgeStageConfig`. The fused `asap_edge`
emit now writes `format: intchunk` + `coldpart_endpoint:` into the agent
`cold:` block only when the format is intchunk; the default fragment
path emits neither key, leaving the cold block byte-identical to today.

Operators opt in per deployment via the `ASAP_COLD_FORMAT=intchunk` env
knob (mirrors how `ASAP_CLUSTER` drives the cold external-label default),
read in the shared bootstrap/replan stitch. The coldpart endpoint is
derived from the fragment ship endpoint (same merger host:port,
`/ingest/coldpart` path) unless overridden.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 5f1cd5f into main May 25, 2026
@zzylol
zzylol deleted the feat/cold-archive-format-emit branch July 17, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant