Skip to content

asap_sketchlib ↔ sketchlib-go byte-format harmonization (cross-language parity) #243

Description

@zzylol

Background

PR #242 (Phase 3 step 2) added a cross-language byte-parity harness comparing envelopes produced by `asap-precompute-go` (Go runtime + sketchlib-go) vs `asap-precompute-rs` (Rust runtime + asap_sketchlib). All 5 sketches diverge byte-for-byte; tests are honestly `#[ignore]`'d in `asap-precompute-rs/tests/cross_language_parity.rs` with documented reasons.

This issue tracks the asap_sketchlib + sketchlib-go work needed to close the gap.

Per-sketch divergences

  • DDSketch: `asap_sketchlib::DdSketch.store_counts` includes 128-aligned padding zeros; `sketchlib-go` emits a tight bucket array.
  • KLL: `asap_sketchlib::KLL` doesn't expose `levels()` / `items()` accessors; the Rust wrapper builds `KllState.items` from a copy-on-update history vec, which diverges from sketchlib-go's compactor view.
  • HLL: `asap_sketchlib::HllSketch::update` uses a different hash seed path than `sketchlib-go::HyperLogLog::Update`; registers populated differently.
  • CountSketch: `asap_sketchlib` uses `twox_hash::XxHash64::oneshot(r, key)`; `sketchlib-go` uses `DeriveIndex/DeriveSign` over a seeded HashSpec table.
  • CountMinSketch: same hash divergence as CountSketch.

asap_sketchlib API gaps

  • No `serialize_portable()` / `serialize_proto_bytes()` helpers — wrappers in `asap-precompute-rs/src/sketches/` build the proto manually.
  • No `compute_delta(prev, current, threshold)` — wrappers force `is_full=true`, bandwidth-inefficient vs Go.
  • No `KLL::levels()` / `KLL::items()` accessors.
  • No hash-layer compatibility shim with `sketchlib-go`'s `CanonicalHashSeed` table.

Why this matters

ADR-0002 promises bit-identical wire format across runtimes. Without harmonization, a fleet mixing Go and Rust ASAP edge agents would produce envelopes that backends cannot merge cross-source.

Scope

  • Multiple PRs across `sketchlib-go` and `asap_sketchlib` (and a follow-up in this repo to un-`#[ignore]` the parity tests).
  • Deferred until there's a real Rust-based edge agent consumer (Vector adapter, OTAP-Rust receiver). For now, Go-only fleets are unaffected.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions