Skip to content

feat(ingest): consume asap-precompute-rs for envelope/delta/merge logic - #76

Merged
zzylol merged 1 commit into
mainfrom
phase3/backend-consumes-asap-precompute-rs
May 5, 2026
Merged

zzylol merged 1 commit into
mainfrom
phase3/backend-consumes-asap-precompute-rs

Conversation

@zzylol

@zzylol zzylol commented May 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Phase 3 step 3 of the ASAP edge-framework migration. Backend ingest path now delegates the shared envelope-parsing, sketch-reconstruction, and merge logic to asap-precompute-rs (the host-neutral Rust edge runtime). The same code now runs in agents and the backend.
  • New module asap-query-engine/src/precompute_operators/edge_runtime_adapter.rs re-exports asap-precompute-rs's runtime view types (SketchEnvelope, Encoding, SketchType, the Sketch trait family) and exposes reconstruct_via_runtime / unwrap_envelope_state / encode_ddsketch_envelope / merge_ddsketches_via_runtime helpers used by the decode_modified_otlp_sketch_bytes hot path.
  • DDSketch and KLL byte parity is live (upstream asap_sketchlib PRs feat(sketch-db): create_checked retention validation (Method B) #40, feat(sketch-db): Phase 5 eviction — SchemaEvictionService background task #41). HLL / CountSketch / CountMinSketch byte parity is tracked under ProjectASAP/ASAPCollector#243; their tests are gated #[ignore = "blocked on ASAPCollector#243 HLL/CS/CMS byte parity"].
  • Query-side engine is unchanged: PromQL aggregation, storage, and query planning all stay in this repo. Backend's per-accumulator AggregateCore / query_statistic / MergeableAccumulator traits stay; sparse-delta apply (apply_modified_otlp_delta_bytes) stays.

Notable details

  • DDSketch / KLL ingest paths in decode_modified_otlp_sketch_bytes try the runtime adapter first (envelope-wrapped wire format, the canonical edge-framework shape). On envelope-decode failure (bare *State proto, as some unit-test / pre-envelope payloads still emit per the PR fix(ingest): decode envelope-wrapped CountMinState from DC processor #14 contract on from_sketchlib_proto_bytes), the dispatcher falls back to the backend's existing native decoder so existing e2e tests keep working unchanged.
  • HLL / CountSketch / CountMinSketch ingest still uses backend's per-accumulator decoders until refactor(emit): re-target agent OTLP export from gateway:4317 → backend:4317 #243 lands.
  • Workspace Cargo.toml adds a [patch."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/ProjectASAP/asap_sketchlib"] block pointing at the local ../asap_sketchlib checkout. This is required so the asap_sketchlib::sketches::* types unify across the two consumers (backend's git-sourced dep and asap-precompute-rs's path-sourced dep) at the FFI boundary (wrapper.inner() → backend accumulator).
  • asap-precompute-rs is consumed via a sibling path dep (../../ASAPCollector/asap-precompute-rs) — see the comment in asap-query-engine/Cargo.toml for why git-source resolution doesn't work today (parent-repo submodule pins).

Test plan

  • cargo build --workspace (succeeds)
  • cargo test --workspace --no-run (all test binaries compile)
  • cargo test --test edge_runtime_consumes_precompute_rs (6 passed, 3 ignored)
  • cargo test --test e2e_modified_otlp_sketch_path (6/6 passed; DDSketch + KLL routes through new adapter, others use backend's native decoder)
  • cargo test --lib precompute_operators::edge_runtime_adapter (3 unit tests pass: round-trip, merge, envelope unwrap)
  • HLL / CountSketch / CountMinSketch round-trip tests are present and #[ignore]'d pending ASAPCollector#243.

Note: the workspace lib has 35 pre-existing test failures in tests::datafusion::* / tests::elastic_* / tests::prometheus_forwarding_tests / tests::schema_timeline_dispatch_tests paths. These exist on origin/main and are unrelated to this PR's ingest/adapter work.

ASAPQuery-backend ingest path now uses asap-precompute-rs (Rust edge
runtime) for shared logic: envelope parsing, delta application, sketch
state reconstruction, merge. Query-side engine (PromQL aggregation,
storage, query planning) stays in this repo.

Phase 3 step 3 of the ASAP edge-framework migration. DDSketch + KLL
byte parity on upstream asap_sketchlib (#40, #41). HLL/CountSketch/CMS
still pending — functional tests gated with #[ignore = "blocked on
ASAPCollector#243"].

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 67c1d99 into main May 5, 2026
@zzylol
zzylol deleted the phase3/backend-consumes-asap-precompute-rs branch May 5, 2026 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant