Skip to content

fix(edge): P0/P1/P2 code-review fixes across gorilla-go, precompute-go, asapedgeprocessor (+Rust parity, docs) - #451

Merged
zzylol merged 3 commits into
mainfrom
fix/edge-review-p0p1p2
May 26, 2026
Merged

zzylol merged 3 commits into
mainfrom
fix/edge-review-p0p1p2

Conversation

@zzylol

@zzylol zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor

Implements the P0/P1/P2 fixes from a code review of the edge stack (asap-gorilla-go, asap-precompute-go, asapedgeprocessor), plus Rust byte-parity reconciliation, the B6 fix in both runtimes, and a delta-baseline design doc. ASAPQuery-backend needs no code changes (verified against its current main).

P0 — correctness / silent data loss

  • HLL & CountSketch full-state ApplyDelta (asap-precompute-go/sketches/{hll,countsketch}.go): tried delta-decode first, so a full-state envelope silently merged to 0. Now full-state-first (mirrors CMS). Real-wrapper regression test added.
  • Unsupported metric types no longer dropped (asapedgeprocessor/ingest.go): Histogram/Summary/ExponentialHistogram were silently dropped from passthrough and never archived. Now always forwarded raw + counted.
  • Unbounded maps capped (warm_sketch.go/warm_sum.go): wired MaxSeries/OnOverflow (+ sum-group cap), default 100000.
  • Shutdown durable drain (processor.go): no longer skips cold-part/spool drain when the flush-wait ctx expires.
  • CMS dim validation (sketches/cms.go): non-pow2 cols panicked on query; now cols→next pow2 + rows clamped to the 64-bit row-hash budget. Mirrored into Rust for asap_sketchlib ↔ sketchlib-go byte-format harmonization (cross-language parity) #243 parity.
  • OOODrop per-fragment delta (asap-gorilla-go/fragment.go): cumulative counter was summed per fragment (~N·D over-count); now a per-flush delta.

P1 — designed but missing

  • Control-plane wiring (asapedgeprocessor/control_plane.go): optional control_channel block; Start() polls and applies config via Precompute.UpdateConfig in-place. Disabled by default.
  • Delta transmission wired per-family — default OFF (see backend-compat + design note below).
  • AllowedLateness set to match cold ReorderGrace.
  • Regex label matchers (precompute-go/matchers.go): =~/!~, Prometheus-anchored.
  • Sliding windows (precompute-go/window.go) implemented, with a guardrail doc that they must not feed the additive-merge backend.

P2 — hygiene

gorilla-go: INT decode cursor advance, README rewrite (removed non-existent GORILLA1 API), ASAPCC1/intchunk archive-tier doc, dead code, low-sev guards. precompute-go: quantile clamp, telemetry counters. edge: sum start>end + maxObserved reset, observe-error counter, cold seal/ship dedup, -race concurrent ConsumeMetrics test.

B6 — CountSketch/CMS observation subject (fixed in BOTH runtimes)

  • Go edge (warm_sketch.go): CountSketch now counts the attribute-set (key=AttributesKey, weight 1), and CMS counts the attribute-set via KindBytes — not the degenerate metric-name key.
  • Rust OTAP edge (asap-precompute-rs): SketchObserver::observe(sketch, v)observe(sketch, obs) so the family-specific observer keys itself from obs.labels. This fixes CountSketch (was counting the metric name) and the CMS-records-nothing bug (CMS errored on the OTAP Float-kind path). No Go change, no plugin/lifecycle plumbing; cargo test + --features otap green; stale lifecycle test fixed + attribute-set tests added.

Design decision — sketches carry no metric scalars

Per project direction: no sketch (DDSketch included) carries metric count/sum/min/max in the full proto or the delta proto. Exact aggregates are separate, controller-provisioned exact edge aggregations (e.g. the existing sum family → delta-Sum metric; a MinMax family), never fields on the sketch. This makes delta transmission uniformly sketch-state-only across all families. docs/delta-baseline-contract.md is updated to this model. The actual proto field removal is a queued follow-up effort (see below).

Backend compatibility (ASAPQuery-backend) — no changes required

  • Delta default OFF: backend applies deltas additively onto a running reconstructed base, but the edge resets per tumbling window → naive enablement double/mis-counts. See docs/delta-baseline-contract.md (Option A; uniform sketch-state-only delta; e2e equality-test gate).
  • CMS dims: round cols→pow2 in both Go and Rust so envelopes stay byte-identical (asap_sketchlib ↔ sketchlib-go byte-format harmonization (cross-language parity) #243).
  • Sliding stays edge-tumbling. MaxSeries: backend is eventually-consistent.

Rust parity (#243)

Mirrored CMS/CountSketch dim normalization into asap-precompute-rs; B6 OTAP observer keying fixed there too. (Full-state ApplyDelta needed no Rust change — already correct.)

Verification

  • Go: build/vet/test green across all three modules (incl. the -race test).
  • Rust: cargo build + cargo test (incl. --features otap) green.
  • OCB end-to-end: build_asap_otel.sh produced the full asap-otel collector; asap-otel components lists asap_edge. ✅

Related PRs

Follow-ups (not in this PR)

🤖 Generated with Claude Code

zzylol and others added 2 commits May 26, 2026 06:32
…apedgeprocessor

Agent-implemented batch (pre-reconciliation):
- precompute-go: HLL/CountSketch full-state ApplyDelta, CMS dim validation,
  regex matchers, sliding windows, quantile clamp, telemetry counters
- gorilla-go: OOODrop per-fragment delta, INT decode cursor, README+archive-tier
  docs, dead code, low-sev guards
- asapedgeprocessor: unsupported-type passthrough, MaxSeries/overflow caps,
  shutdown durable drain, control-plane wiring, delta transmission (default OFF),
  AllowedLateness, sum ts fix, observe-error counter, B6 CountSketch subject,
  cold seal dedup, -race ConsumeMetrics test

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- precompute-go: document Sliding window must not feed the additive-merge
  backend (overlapping emissions double-count); correct stale 'tumbling-only'
  comment now that sliding is implemented.
- precompute-rs: mirror Go's CMS/CountSketch dim normalization
  (cols->next_pow2, clamp rows to the 64-bit row-hash budget) so Go and Rust
  emit byte-identical envelopes (#243); cols=2000->2048 on both sides.
- docs: add delta-baseline-contract.md — why edge delta transmission must stay
  OFF until the edge per-window-reset model is reconciled with the backend's
  additive-onto-running-base merge; recommends Option A, DDSketch first.

Backend (ASAPQuery-backend) needs no code changes; delta transmission ships
default-OFF behind the per-family flag.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol

zzylol commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

Companion PRs (both open):

End-to-end verified: OCB build_asap_otel.sh produced the full asap-otel collector binary with the fixed source pinned via local replace; asap-otel components lists asap_edge.

…etch/CMS)

Change SketchObserver::observe(sketch, v) -> observe(sketch, obs) so the
family-specific observer keys itself from obs.labels (the window.rs call site
already has obs in scope). No Go change, no plugin/lifecycle plumbing.

- CMSObserver: drop the hard 'expected Bytes' error; key = utf8(bytes) if
  present else attributes_key(labels) -> fixes CMS-records-nothing on the
  OTAP Float-kind path.
- CountSketchObserver: key = bytes else attributes_key(labels) else default_key;
  weight 1 on the labels path -> counts per-attribute-set frequency like the Go
  edge, not the degenerate metric-name key.
- DDSketch/KLL/HLL observers + 2 test mocks: mechanical (read obs.value.*).
- Fix stale otap_lifecycle CMS test (now emits) + add attribute-set tests.
- cargo test + --features otap green.

docs/delta-baseline-contract.md: correct the now-invalid 'DDSketch carries
explicit scalars' rationale — per the design rule, sketches carry NO
count/sum/min/max (full or delta); exact aggregates are separate
controller-provisioned edge aggregations; delta is uniformly sketch-state-only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit 3e80523 into main May 26, 2026
@zzylol
zzylol deleted the fix/edge-review-p0p1p2 branch May 26, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant