feat(asap-precompute-rs): runtime port + sketch wrappers + cross-language parity (Phase 3 step 2) - #242
Merged
Conversation
zzylol
force-pushed
the
phase3/precompute-rs-state-machine
branch
from
May 4, 2026 21:29
45d80be to
209621d
Compare
…uage parity (Phase 3 step 2)
Three pieces of work in one commit:
1. Runtime port (Rust mirror of asap-precompute-go):
Replace unimplemented!() stubs with Rust ports of the Go runtime —
WindowState, SeriesEntry, SnapshotCache::compute_delta
(always-refresh), PrecomputeImpl::observe / observe_envelope /
tick / drain. Byte-format invariants preserved: SeriesKey /
AttributesKey output, delta-cache always-refresh, Drain
unconditional rotation.
2. Real sketch wrappers in asap-precompute-rs/src/sketches/:
{ddsketch,kll,hll,countsketch,cms}.rs — wrappers over
asap_sketchlib's wire-format-aligned types implementing the
Sketch trait family. Mirrors asap-precompute-go/sketches/.
Each wrapper provides constructor + update + snapshot
(proto-encoded SketchEnvelope) + apply_delta + merge + reset.
Five new integration tests in tests/runtime.rs exercise
observe → tick → envelope output per wrapper.
Trait extension: Sketch::as_any_mut() added so paired observers
can downcast to the concrete wrapper. Default impl on FakeSketch
in tests/runtime.rs preserves backwards compatibility.
3. Cross-language byte-parity harness:
- integration/parity/golden_test.go generates per-sketch
fixtures via sketchlib-go's portable serializers
(SerializePortable / SerializeProtoBytes / SerializeProtoBytesFO).
Run with GOLDEN_REGEN=1 to refresh.
- asap-precompute-rs/tests/cross_language_parity.rs loads each
fixture and asserts byte-equality against the Rust wrapper's
output. Five parity tests are #[ignore] with documented
reasons — asap_sketchlib's current API surface causes wire
bytes to diverge from sketchlib-go for every sketch type
(different bucket-store layouts, missing serialize helpers,
diverging hash-seed paths). Two sanity tests verify the
fixture wiring.
Naming cleanup: rename "state machine" → "runtime" everywhere to
match asap-precompute-go's package-doc terminology
("host-neutral edge precompute runtime"). Branch name kept as-is.
ADR-0002 §"Performance contract" updated: Rust runtime mirrors
asap-precompute-go's runtime, with edge-only framing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
force-pushed
the
phase3/precompute-rs-state-machine
branch
from
May 4, 2026 21:37
209621d to
94c8eec
Compare
4 tasks
zzylol
added a commit
that referenced
this pull request
May 5, 2026
…rs (#246) PR #242 over-deleted ADR text when fixing edge/backend terminology. The correct scoping is: - Backend QUERY-side engine (PromQL aggregation, storage, planning): separate design, not shared with edge runtime. - Backend INGEST path (envelope parsing, delta apply, sketch reconstruction, merge): SHARED via asap-precompute-rs. Restores §"What gets extracted" / §"Positive consequences" framing. References issue #243 as byte-alignment prerequisite for backend integration. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 15, 2026
…pquery-backend (#377) The default data path is now: fake-exporter → agent → asapquery-backend collapsing the former: fake-exporter → agent → gateway → asapquery-backend Why this works: asapquery-backend's `--enable-otel-ingest` already accepts sketch data points and merges them per-aggregation_id via its precompute engine accumulators (DDSketchAccumulator / HLLAccumulator / etc.) — no middle-tier OTel gateway merge processor is required. The asapquery-backend control-plane PRs #241 / #242 / #243 retire the gateway-targeted emit pieces (legacy backend-collector emitter + shadow `BackendCollectorConfig` + agent OTLP default_host). Code kept-in-source per "keep gateway impl in repos" intent: - All `configs/gateway*.yaml` files preserved - `configs/asap/asap-otel-gateway-mvp-placeholder.yaml` preserved - Gateway service definition in `base.yml` profile-gated under `["gateway-legacy"]` — `docker compose --profile gateway-legacy …` still spins it up - `gateway_up` / `gateway_down` functions in multinode `run_demo.sh` defined-but-uncalled (lifecycle dropped from `arm_up` / `arm_down`) Files touched: Agent OTel configs (rename `otlp/gateway` exporter block to `otlp/backend`, change `endpoint: gateway:4317` → `endpoint: backend:4317`): - mvp-singlenode/configs/asap-otel-agent-{b0a-raw-stream, b0b-raw-batched, b2-full, b3-delta, b4-tunable, b6-asap-single-sketch}.yaml - mvp-multinode/configs/asap/asap-otel-agent-b6-asap-single-sketch.yaml Workload tags (assign_to_role: backend|gateway → agent): - mvp-singlenode/configs/workloads.yaml (2 rows) - mvp-singlenode/configs/mvp-workload.yaml (1 row) - mvp-multinode/configs/asap/mvp-workload.yaml (1 row) Compose: - mvp-singlenode/docker-compose/base.yml — gateway service `profiles: ["gateway-legacy"]`; fake-exporter default EXPORTER_TARGET=backend:4317; fake-exporter depends_on→backend - mvp-singlenode/docker-compose/e2e-overlay.yml — gateway override also `profiles: ["gateway-legacy"]` - mvp-singlenode/docker-compose/agents-N{1,10,100}.yml — agent depends_on→backend (was gateway) - mvp-singlenode/docker-compose/gen-agents.sh — template emits same - mvp-singlenode/docker-compose/mvp-multi-stage.yml — gateway override `profiles: ["gateway-legacy"]`; comment notes how to re-enable multi-stage flow (requires `--profile gateway-legacy` + gateway- targeting AGENT_CONFIG) Multinode orchestration: - mvp-multinode/topology.env — NODE1 reclassified "unused / reserved" (DNS alias preserved for opt-in) - mvp-multinode/scripts/run_demo.sh — gateway_up/down dropped from arm lifecycle (functions remain) - mvp-multinode/scripts/run_demo_sweep.sh — roles map node1 "gateway" → "unused" Docs: - mvp-singlenode/README.md — port table + topology blurb updated - mvp-multinode/README.md — topology diagram updated Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three pieces of work in one commit:
asap-precompute-go) — replaces everyunimplemented!()stub left by Phase 3 step 1 (PR feat(asap-precompute-rs): bootstrap (Phase 3 step 1) #241). Translation,not redesign.
asap-precompute-rs/src/sketches/overasap_sketchlib's wire-format-aligned types — mirrorsasap-precompute-go/sketches/.in
integration/parity/golden/, Rust tests inasap-precompute-rs/tests/cross_language_parity.rs. Honest results:bytes diverge for every sketch today; 5 tests
#[ignore]withdocumented reasons.
Naming cleanup: rename "state machine" → "runtime" everywhere
(matches
asap-precompute-go's package-doc: "host-neutral edgeprecompute runtime"). Branch name kept as-is.
ADR-0002 fix: §"Performance contract" Rust bullet rewritten to drop
"bit-identical to today's per-accumulator
apply_proto_delta_bytes"language; new phrasing pins the contract to mirroring
asap-precompute-go's runtime.Port table
src/window.rsasap-precompute-go/window.gosrc/snapshot_cache.rsasap-precompute-go/snapshot_cache.gosrc/precompute.rs::PrecomputeImpl::{observe, observe_envelope, tick, drain}asap-precompute-go/precompute.gosrc/sketches/{ddsketch,kll,hll,countsketch,cms}.rsasap-precompute-go/sketches/{ddsketch,kll,hll,countsketch,cms}.goTrait surface
The
Sketchtrait gains one method:as_any_mut(&mut self) -> &mut dyn Any,implemented by every wrapper as
{ self }. This unblocks pairedSketchObserverimpls that need to call concrete-type methods likeupdate(f64)after the runtime hands them a&mut dyn Sketch. TheFakeSketchtest double intests/runtime.rsadds the same{ self }impl; no other downstream changes.Sketch wrapper status
Each wrapper:
pub fn new(params...) -> SelfSketch(snapshot, compute_delta_against, apply_delta, merge, reset)QuantileSketch::quantile,CardinalitySketch::estimate_cardinality,FrequencySketch::estimate_count+top_k)prost::Message::encodeon aSketchEnvelopewiththe inner state proto (
DdSketchState/KllState/HyperLogLogState/CountSketchState/CountMinState).KLL::init_kll_with_seed.compute_delta_againstis pinned tois_full = truefor everywrapper.
asap_sketchlibdoes not currently exposeComputeDeltahelpers (Go'ssketchlib-godoes:ddsketch.ComputeDelta,hll.ComputeRegisterDelta,countsketch.ComputeDelta,cms.ComputeDelta). Until those landupstream, the wrapper emits
ProtoFullenvelopes every window —correct but bandwidth-inefficient versus Go.
Cross-language byte-parity findings
All five
#[ignore]tests fail honestly when run with--include-ignored. Documented per-sketch reasons:asap_sketchlib::DdSketch's bucket-store layout(
Vec<u64>, auto-grown in chunks of 128) emits trailing zerosin
store_countsthatsketchlib-godoesn't. Need eitherstore-growth alignment or a normalization helper.
KllState.itemsfield is built from acopy-on-update history vec, not the compactor's
items[]+levels[]view.asap_sketchlib::KLLdoesn't exposelevels()/items()accessors.asap_sketchlib::HllSketch::updatehashes inputdifferently from
sketchlib-go::HyperLogLog::Update(differenthash seed paths). Registers populated by 50 sequential bytes
diverge register-for-register.
asap_sketchlib::CountSketch::updateusestwox_hash::XxHash64::oneshot(r as u64, key), whilesketchlib-go::CountSketchroutes throughDeriveIndex/DeriveSign. Matrix populated cell-for-cell differently.asap_sketchlibAPI gaps surfaced — needed for Go-parity:DdSketch::serialize_portable() -> Vec<u8>(full envelope)compute_delta(prev: &DdSketch, current: &DdSketch, threshold: u64) -> Vec<u8>for DDSketch / CMS / CountSketch / HLLKLL::levels() -> &[u32]+KLL::items() -> &[f64]accessors sothe wire-format
KllStatecan be built faithfullyxxh3_64with sketchlib-go'sCanonicalHashSeedtable) for HLL / CountSketch / CMSThese are flagged inline in each wrapper's
compute_delta_againstcomment and in the cross-parity test ignore strings. Closing them
is a separate follow-up that touches
asap_sketchlibupstream.Tests (88 total)
precompute / sketches::{ddsketch,kll,hll,countsketch,cms})
Go side:
integration/parity/golden_test.gois deterministic acrossruns (single-series-per-sketch construction); existing
TestParity_AllSketchesstill passes.Validation
cd asap-precompute-rs && cargo build— cleancargo test— 88 tests pass (5 honestly-ignored byte-parity)cargo clippy --all-targets -- -D warnings— cleancargo fmt --check— cleancd integration/parity && go test ./...— pass (existingparity harness + new golden generator)
cd asap-precompute-go && go test ./...— pass🤖 Generated with Claude Code