Skip to content

deploy(mvp-multinode): activate intchunk cold path + sampling experiment knobs - #454

Closed
zzylol wants to merge 1 commit into
mainfrom
chore/node-split-deploy-config
Closed

zzylol wants to merge 1 commit into
mainfrom
chore/node-split-deploy-config

Conversation

@zzylol

@zzylol zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor

Salvages the genuinely-unique, still-applicable experiment config from the stale local branch chore/split-cold-backend-node1 (tip 944e054, snapshotted on the now ~6-PRs-behind base 2939d23). That branch's ~5,300-line WIP is almost entirely already on main via merged PRs, so a direct PR from it would revert merged work. This PR carries only the 3 deploy-config deltas that enable already-merged features not yet turned on in the multinode demo. Config-only: no build required.

Included (each activates a now-on-main feature)

File Change Activates
deploy/mvp-multinode/configs/asap/asap-otel-agent-b6-asap-single-sketch.yaml Add cold.format: intchunk + coldpart_endpoint: .../ingest/coldpart under asap_edge The intchunk decode-on-read cold path (opt-in, landed via #439/#442). The asapedgeprocessor ColdConfig on main already supports format/coldpart_endpoint (config.go); this enables it in the static reference agent config. shard_count left at main's smooth-flush 12 (NOT reverted to the branch's 4).
deploy/mvp-multinode/configs/asap/mvp-workload.yaml Add sample_p: 0.5 to the HLL (unique_users_per_min, #6) and CountMinSketch (endpoint_request_freq, #8) families The fused processor's sampling path (SampleP mapstructure key is already on main; backend rescales x1/p). The workload just activates it.
deploy/mvp-multinode/scripts/run_demo.sh Add -e ASAP_COLD_FORMAT=intchunk and -e ASAP_AGENT_MEMORY_LIMIT_MIB=8192 to the control-plane container The controller-emitted (OpAMP) agent config: ASAP_COLD_FORMAT flips the generated cold format to intchunk and ASAP_AGENT_MEMORY_LIMIT_MIB sets the agent memory limit at emit time. Both env knobs are already consumed by the merged control_plane (emit/mod.rs, emit/stage_config.rs).

Total diff: 3 files, +10 lines, additions only. All three files validated (YAML parses, bash -n clean).

Excluded as superseded (NOT carried — would revert merged work)

Conservative note for reviewer: ASAP_COLD_FORMAT=intchunk (controller-emitted config) and the static format: intchunk in the b6 yaml both express the same intent on their respective config-delivery paths (OpAMP-pushed fused config vs. static reference file). If you only want one activation path, drop whichever you don't use. shard_count is intentionally kept at main's 12.

🤖 Generated with Claude Code

…ent knobs

Salvages the genuinely-unique, still-applicable experiment config from the
stale local node-split branch (chore/split-cold-backend-node1, tip 944e054,
based on the now ~6-PRs-behind 2939d23). All of that branch's code work is
already on main via merged PRs, so this carries ONLY the three deploy-config
deltas that turn on already-merged features and are not yet enabled in the
multinode demo.

Included (config-only, no build):
- asap-otel-agent-b6-asap-single-sketch.yaml: add cold `format: intchunk` +
  `coldpart_endpoint` under the asap_edge cold tier, switching the static
  reference agent config from the gorilla-XOR fragment batch to the lossless
  intchunk decode-on-read cold part (opt-in path landed via #439/#442; the
  processor already supports cold.format/coldpart_endpoint, this just enables
  it). shard_count is left at main's smooth-flush value of 12.
- mvp-workload.yaml: set `sample_p: 0.5` on the HLL (#6 unique_users_per_min)
  and CountMinSketch (#8 endpoint_request_freq) families to exercise the
  fused processor's sampling path (backend rescales x1/p). The feature is
  merged; this workload just activates it.
- run_demo.sh: add `-e ASAP_COLD_FORMAT=intchunk` and
  `-e ASAP_AGENT_MEMORY_LIMIT_MIB=8192` to the control-plane container so the
  controller-emitted (OpAMP) agent config picks up the intchunk cold path and
  the bumped agent memory limit (both env knobs are already consumed by the
  merged control_plane: emit/mod.rs + emit/stage_config.rs).

Excluded as already superseded on main (NOT carried, to avoid reverting
merged work): all asap-precompute-rs changes (Rust sketchlib migration already
on main); asap-gorilla-go coldpart/intchunk/fragment + asap-precompute-go
sampling (#434/#439/#442); the old-layout asapedgeprocessor coldpart_path.go
and sibling edits (pre-#448 god-object layout; intchunk cold ship now lives in
the #448-split processor); and the stale topology.env / run_demo.sh hunks that
predate main's otel-app rename, raw-buffer default, 4096 buffer, shard_count=12
smooth-flush and sketch-persistence flags.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol

zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

Closing per review — demo/experiment config knobs (intchunk cold-path activation, sample_p=0.5 on HLL/CMS, agent memory cap) that only activate already-merged features. No capability lost; the WIP snapshot remains on chore/split-cold-backend-node1 if wanted later.

@zzylol zzylol closed this May 26, 2026
@zzylol
zzylol deleted the chore/node-split-deploy-config branch May 26, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant