Skip to content

feat(asap-precompute-rs): runtime port + sketch wrappers + cross-language parity (Phase 3 step 2) - #242

Merged
zzylol merged 1 commit into
mainfrom
phase3/precompute-rs-state-machine
May 5, 2026
Merged

zzylol merged 1 commit into
mainfrom
phase3/precompute-rs-state-machine

Conversation

@zzylol

@zzylol zzylol commented May 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Three pieces of work in one commit:

  1. Runtime port (Rust mirror of asap-precompute-go) — replaces every
    unimplemented!() stub left by Phase 3 step 1 (PR feat(asap-precompute-rs): bootstrap (Phase 3 step 1) #241). Translation,
    not redesign.
  2. Real sketch wrappers in asap-precompute-rs/src/sketches/ over
    asap_sketchlib's wire-format-aligned types — mirrors
    asap-precompute-go/sketches/.
  3. Cross-language byte-parity harness — Go-generated golden fixtures
    in integration/parity/golden/, Rust tests in
    asap-precompute-rs/tests/cross_language_parity.rs. Honest results:
    bytes diverge for every sketch today; 5 tests #[ignore] with
    documented reasons.

Naming cleanup: rename "state machine" → "runtime" everywhere
(matches asap-precompute-go's package-doc: "host-neutral edge
precompute runtime"). Branch name kept as-is.

ADR-0002 fix: §"Performance contract" Rust bullet rewritten to drop
"bit-identical to today's per-accumulator apply_proto_delta_bytes"
language; new phrasing pins the contract to mirroring
asap-precompute-go's runtime.

Port table

Rust file Go reference
src/window.rs asap-precompute-go/window.go
src/snapshot_cache.rs asap-precompute-go/snapshot_cache.go
src/precompute.rs::PrecomputeImpl::{observe, observe_envelope, tick, drain} asap-precompute-go/precompute.go
src/sketches/{ddsketch,kll,hll,countsketch,cms}.rs asap-precompute-go/sketches/{ddsketch,kll,hll,countsketch,cms}.go

Trait surface

The Sketch trait gains one method: as_any_mut(&mut self) -> &mut dyn Any,
implemented by every wrapper as { self }. This unblocks paired
SketchObserver impls that need to call concrete-type methods like
update(f64) after the runtime hands them a &mut dyn Sketch. The
FakeSketch test double in tests/runtime.rs adds the same
{ self } impl; no other downstream changes.

Sketch wrapper status

Each wrapper:

  • Constructor: pub fn new(params...) -> Self
  • Implements Sketch (snapshot, compute_delta_against, apply_delta, merge, reset)
  • Implements the appropriate sub-trait (QuantileSketch::quantile,
    CardinalitySketch::estimate_cardinality,
    FrequencySketch::estimate_count + top_k)
  • Snapshot uses prost::Message::encode on a SketchEnvelope with
    the inner state proto (DdSketchState / KllState /
    HyperLogLogState / CountSketchState / CountMinState).
  • KLL takes a deterministic seed via KLL::init_kll_with_seed.

compute_delta_against is pinned to is_full = true for every
wrapper.
asap_sketchlib does not currently expose
ComputeDelta helpers (Go's sketchlib-go does:
ddsketch.ComputeDelta, hll.ComputeRegisterDelta,
countsketch.ComputeDelta, cms.ComputeDelta). Until those land
upstream, the wrapper emits ProtoFull envelopes every window —
correct but bandwidth-inefficient versus Go.

Cross-language byte-parity findings

All five #[ignore] tests fail honestly when run with
--include-ignored. Documented per-sketch reasons:

  • DDSketch: asap_sketchlib::DdSketch's bucket-store layout
    (Vec<u64>, auto-grown in chunks of 128) emits trailing zeros
    in store_counts that sketchlib-go doesn't. Need either
    store-growth alignment or a normalization helper.
  • KLL: the wrapper's KllState.items field is built from a
    copy-on-update history vec, not the compactor's
    items[]+levels[] view. asap_sketchlib::KLL doesn't expose
    levels() / items() accessors.
  • HLL: asap_sketchlib::HllSketch::update hashes input
    differently from sketchlib-go::HyperLogLog::Update (different
    hash seed paths). Registers populated by 50 sequential bytes
    diverge register-for-register.
  • CountSketch: asap_sketchlib::CountSketch::update uses
    twox_hash::XxHash64::oneshot(r as u64, key), while
    sketchlib-go::CountSketch routes through DeriveIndex /
    DeriveSign. Matrix populated cell-for-cell differently.
  • CMS: same hashing-divergence pattern as CountSketch.

asap_sketchlib API gaps surfaced — needed for Go-parity:

  • DdSketch::serialize_portable() -> Vec<u8> (full envelope)
  • compute_delta(prev: &DdSketch, current: &DdSketch, threshold: u64) -> Vec<u8> for DDSketch / CMS / CountSketch / HLL
  • KLL::levels() -> &[u32] + KLL::items() -> &[f64] accessors so
    the wire-format KllState can be built faithfully
  • Hash-layer compatibility shim (xxh3_64 with sketchlib-go's
    CanonicalHashSeed table) for HLL / CountSketch / CMS

These are flagged inline in each wrapper's compute_delta_against
comment and in the cross-parity test ignore strings. Closing them
is a separate follow-up that touches asap_sketchlib upstream.

Tests (88 total)

  • 43 lib unit tests (config / matchers / window / snapshot_cache /
    precompute / sketches::{ddsketch,kll,hll,countsketch,cms})
  • 11 api_surface tests
  • 27 runtime integration tests (22 fake-sketch + 5 real-sketch)
  • 7 cross-language parity tests (2 sanity + 5 ignored byte-parity)
  • 0 ignored except the 5 documented byte-parity gaps

Go side: integration/parity/golden_test.go is deterministic across
runs (single-series-per-sketch construction); existing
TestParity_AllSketches still passes.

Validation

  • cd asap-precompute-rs && cargo build — clean
  • cargo test — 88 tests pass (5 honestly-ignored byte-parity)
  • cargo clippy --all-targets -- -D warnings — clean
  • cargo fmt --check — clean
  • cd integration/parity && go test ./... — pass (existing
    parity harness + new golden generator)
  • cd asap-precompute-go && go test ./... — pass

🤖 Generated with Claude Code

@zzylol
zzylol force-pushed the phase3/precompute-rs-state-machine branch from 45d80be to 209621d Compare May 4, 2026 21:29
@zzylol zzylol changed the title feat(asap-precompute-rs): state machine port + ADR-0002 fix (Phase 3 step 2) feat(asap-precompute-rs): runtime port + sketch wrappers + cross-language parity (Phase 3 step 2) May 4, 2026
…uage parity (Phase 3 step 2)

Three pieces of work in one commit:

1. Runtime port (Rust mirror of asap-precompute-go):
   Replace unimplemented!() stubs with Rust ports of the Go runtime —
   WindowState, SeriesEntry, SnapshotCache::compute_delta
   (always-refresh), PrecomputeImpl::observe / observe_envelope /
   tick / drain. Byte-format invariants preserved: SeriesKey /
   AttributesKey output, delta-cache always-refresh, Drain
   unconditional rotation.

2. Real sketch wrappers in asap-precompute-rs/src/sketches/:
   {ddsketch,kll,hll,countsketch,cms}.rs — wrappers over
   asap_sketchlib's wire-format-aligned types implementing the
   Sketch trait family. Mirrors asap-precompute-go/sketches/.
   Each wrapper provides constructor + update + snapshot
   (proto-encoded SketchEnvelope) + apply_delta + merge + reset.
   Five new integration tests in tests/runtime.rs exercise
   observe → tick → envelope output per wrapper.

   Trait extension: Sketch::as_any_mut() added so paired observers
   can downcast to the concrete wrapper. Default impl on FakeSketch
   in tests/runtime.rs preserves backwards compatibility.

3. Cross-language byte-parity harness:
   - integration/parity/golden_test.go generates per-sketch
     fixtures via sketchlib-go's portable serializers
     (SerializePortable / SerializeProtoBytes / SerializeProtoBytesFO).
     Run with GOLDEN_REGEN=1 to refresh.
   - asap-precompute-rs/tests/cross_language_parity.rs loads each
     fixture and asserts byte-equality against the Rust wrapper's
     output. Five parity tests are #[ignore] with documented
     reasons — asap_sketchlib's current API surface causes wire
     bytes to diverge from sketchlib-go for every sketch type
     (different bucket-store layouts, missing serialize helpers,
     diverging hash-seed paths). Two sanity tests verify the
     fixture wiring.

Naming cleanup: rename "state machine" → "runtime" everywhere to
match asap-precompute-go's package-doc terminology
("host-neutral edge precompute runtime"). Branch name kept as-is.

ADR-0002 §"Performance contract" updated: Rust runtime mirrors
asap-precompute-go's runtime, with edge-only framing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol force-pushed the phase3/precompute-rs-state-machine branch from 209621d to 94c8eec Compare May 4, 2026 21:37
@zzylol
zzylol merged commit 9e2384c into main May 5, 2026
@zzylol
zzylol deleted the phase3/precompute-rs-state-machine branch May 5, 2026 01:16
zzylol added a commit that referenced this pull request May 5, 2026
…rs (#246)

PR #242 over-deleted ADR text when fixing edge/backend terminology.
The correct scoping is:
- Backend QUERY-side engine (PromQL aggregation, storage, planning):
  separate design, not shared with edge runtime.
- Backend INGEST path (envelope parsing, delta apply, sketch
  reconstruction, merge): SHARED via asap-precompute-rs.

Restores §"What gets extracted" / §"Positive consequences" framing.
References issue #243 as byte-alignment prerequisite for backend
integration.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol added a commit that referenced this pull request May 15, 2026
…pquery-backend (#377)

The default data path is now:
  fake-exporter → agent → asapquery-backend
collapsing the former:
  fake-exporter → agent → gateway → asapquery-backend

Why this works: asapquery-backend's `--enable-otel-ingest` already
accepts sketch data points and merges them per-aggregation_id via its
precompute engine accumulators (DDSketchAccumulator / HLLAccumulator /
etc.) — no middle-tier OTel gateway merge processor is required. The
asapquery-backend control-plane PRs #241 / #242 / #243 retire the
gateway-targeted emit pieces (legacy backend-collector emitter +
shadow `BackendCollectorConfig` + agent OTLP default_host).

Code kept-in-source per "keep gateway impl in repos" intent:
- All `configs/gateway*.yaml` files preserved
- `configs/asap/asap-otel-gateway-mvp-placeholder.yaml` preserved
- Gateway service definition in `base.yml` profile-gated under
  `["gateway-legacy"]` — `docker compose --profile gateway-legacy …`
  still spins it up
- `gateway_up` / `gateway_down` functions in multinode `run_demo.sh`
  defined-but-uncalled (lifecycle dropped from `arm_up` / `arm_down`)

Files touched:

Agent OTel configs (rename `otlp/gateway` exporter block to
`otlp/backend`, change `endpoint: gateway:4317` → `endpoint:
backend:4317`):
- mvp-singlenode/configs/asap-otel-agent-{b0a-raw-stream,
  b0b-raw-batched, b2-full, b3-delta, b4-tunable,
  b6-asap-single-sketch}.yaml
- mvp-multinode/configs/asap/asap-otel-agent-b6-asap-single-sketch.yaml

Workload tags (assign_to_role: backend|gateway → agent):
- mvp-singlenode/configs/workloads.yaml         (2 rows)
- mvp-singlenode/configs/mvp-workload.yaml      (1 row)
- mvp-multinode/configs/asap/mvp-workload.yaml  (1 row)

Compose:
- mvp-singlenode/docker-compose/base.yml — gateway service
  `profiles: ["gateway-legacy"]`; fake-exporter default
  EXPORTER_TARGET=backend:4317; fake-exporter depends_on→backend
- mvp-singlenode/docker-compose/e2e-overlay.yml — gateway override
  also `profiles: ["gateway-legacy"]`
- mvp-singlenode/docker-compose/agents-N{1,10,100}.yml — agent
  depends_on→backend (was gateway)
- mvp-singlenode/docker-compose/gen-agents.sh — template emits same
- mvp-singlenode/docker-compose/mvp-multi-stage.yml — gateway override
  `profiles: ["gateway-legacy"]`; comment notes how to re-enable
  multi-stage flow (requires `--profile gateway-legacy` + gateway-
  targeting AGENT_CONFIG)

Multinode orchestration:
- mvp-multinode/topology.env — NODE1 reclassified "unused / reserved"
  (DNS alias preserved for opt-in)
- mvp-multinode/scripts/run_demo.sh — gateway_up/down dropped from
  arm lifecycle (functions remain)
- mvp-multinode/scripts/run_demo_sweep.sh — roles map node1
  "gateway" → "unused"

Docs:
- mvp-singlenode/README.md — port table + topology blurb updated
- mvp-multinode/README.md — topology diagram updated

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant