Conversation
…ent knobs Salvages the genuinely-unique, still-applicable experiment config from the stale local node-split branch (chore/split-cold-backend-node1, tip 944e054, based on the now ~6-PRs-behind 2939d23). All of that branch's code work is already on main via merged PRs, so this carries ONLY the three deploy-config deltas that turn on already-merged features and are not yet enabled in the multinode demo. Included (config-only, no build): - asap-otel-agent-b6-asap-single-sketch.yaml: add cold `format: intchunk` + `coldpart_endpoint` under the asap_edge cold tier, switching the static reference agent config from the gorilla-XOR fragment batch to the lossless intchunk decode-on-read cold part (opt-in path landed via #439/#442; the processor already supports cold.format/coldpart_endpoint, this just enables it). shard_count is left at main's smooth-flush value of 12. - mvp-workload.yaml: set `sample_p: 0.5` on the HLL (#6 unique_users_per_min) and CountMinSketch (#8 endpoint_request_freq) families to exercise the fused processor's sampling path (backend rescales x1/p). The feature is merged; this workload just activates it. - run_demo.sh: add `-e ASAP_COLD_FORMAT=intchunk` and `-e ASAP_AGENT_MEMORY_LIMIT_MIB=8192` to the control-plane container so the controller-emitted (OpAMP) agent config picks up the intchunk cold path and the bumped agent memory limit (both env knobs are already consumed by the merged control_plane: emit/mod.rs + emit/stage_config.rs). Excluded as already superseded on main (NOT carried, to avoid reverting merged work): all asap-precompute-rs changes (Rust sketchlib migration already on main); asap-gorilla-go coldpart/intchunk/fragment + asap-precompute-go sampling (#434/#439/#442); the old-layout asapedgeprocessor coldpart_path.go and sibling edits (pre-#448 god-object layout; intchunk cold ship now lives in the #448-split processor); and the stale topology.env / run_demo.sh hunks that predate main's otel-app rename, raw-buffer default, 4096 buffer, shard_count=12 smooth-flush and sketch-persistence flags. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Closing per review — demo/experiment config knobs (intchunk cold-path activation, sample_p=0.5 on HLL/CMS, agent memory cap) that only activate already-merged features. No capability lost; the WIP snapshot remains on chore/split-cold-backend-node1 if wanted later. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Salvages the genuinely-unique, still-applicable experiment config from the stale local branch
chore/split-cold-backend-node1(tip944e054, snapshotted on the now ~6-PRs-behind base2939d23). That branch's ~5,300-line WIP is almost entirely already on main via merged PRs, so a direct PR from it would revert merged work. This PR carries only the 3 deploy-config deltas that enable already-merged features not yet turned on in the multinode demo. Config-only: no build required.Included (each activates a now-on-main feature)
deploy/mvp-multinode/configs/asap/asap-otel-agent-b6-asap-single-sketch.yamlcold.format: intchunk+coldpart_endpoint: .../ingest/coldpartunderasap_edgeasapedgeprocessorColdConfigon main already supportsformat/coldpart_endpoint(config.go); this enables it in the static reference agent config.shard_countleft at main's smooth-flush 12 (NOT reverted to the branch's 4).deploy/mvp-multinode/configs/asap/mvp-workload.yamlsample_p: 0.5to the HLL (unique_users_per_min, #6) and CountMinSketch (endpoint_request_freq, #8) familiesSamplePmapstructure key is already on main; backend rescales x1/p). The workload just activates it.deploy/mvp-multinode/scripts/run_demo.sh-e ASAP_COLD_FORMAT=intchunkand-e ASAP_AGENT_MEMORY_LIMIT_MIB=8192to the control-plane containerASAP_COLD_FORMATflips the generated cold format to intchunk andASAP_AGENT_MEMORY_LIMIT_MIBsets the agent memory limit at emit time. Both env knobs are already consumed by the mergedcontrol_plane(emit/mod.rs,emit/stage_config.rs).Total diff: 3 files, +10 lines, additions only. All three files validated (YAML parses,
bash -nclean).Excluded as superseded (NOT carried — would revert merged work)
asap-precompute-rschanges — Rust sketchlib migration is already on main (sketches/cms.rsdiff vs main = 0 lines).asap-gorilla-gocoldpart/intchunk/fragment +asap-precompute-go/sketchessampling — merged via feat(intchunk): cold INT best-of-N lossless value chunk codec #434 / feat: coldpart Part format in asap-gorilla-go + agent intchunk cold-part ship path #439 / perf(asapedgeprocessor): accumulate intchunk cold parts per block_duration #442 and the sample_p merge.asapedgeprocessorcode (coldpart_path.go+ tests, config/factory/sketch/processor edits) — written against the pre-refactor(asapedgeprocessor): split god-object files into warm/cold/ingest/flush units #448 god-object layout atotel-arrow-asap/collector/...; main now has the refactor(asapedgeprocessor): split god-object files into warm/cold/ingest/flush units #448 split atopentelemetry-collector-contrib-patch/processor/asapedgeprocessor/withcold_intchunk_accum.go/cold_intchunk_shipper.goalready shipping the intchunk cold path. Superseded.topology.envhunks — the branch's raw-buffer default +MAX_BUFFER_PER_SERIES=4096are already on main; the branch's version is older (still uses pre-renameEXPORTER_*vars + the droppedEXPORTER_RATEknob + thefake-exporterimage name). Applying it would revert main'sotel-apprename. Nothing unique left.run_demo.shhunks — theotel-apprename,data/sketch-persistencemkdir, and the--persistence-*data-plane flags are already on main; the branch's flag-vs-env producer invocation is the stale pre-rename form. Only the two control-plane env knobs above were genuinely new.🤖 Generated with Claude Code