perf: reduce backend precompute engine CPU

[zzylol]

Path located

The asapquery-backend precompute/ingest path that consumes the agents' warm-tier OTLP output (delta Sums + sketch envelopes) and merges into the sketch store:

• data_plane/src/drivers/ingest/otel.rs — OTLP gRPC/HTTP receiver; route_otlp_to_precompute + route_modified_otlp_sketches_to_precompute run per ingest batch.
• data_plane/src/storage_engines/sketch_db/lifecycle/reconcile.rs — reconcile_from_streaming_config, called from both ingest routines on every batch.
• data_plane/src/storage_engines/sketch_db/index/mod.rs — SketchStore + SketchInstanceMetadata (the cloned type).

Profiling method

Live perf on node2's running asap-backend (PID 429550, ~116% CPU under sustained agent load). perf was absent on node2; installed linux-tools-5.15.0-168-generic, then perf record -F 199 -g -p <pid> -- sleep 30 (6906 samples). The release binary symbolized cleanly.

Top hotspots (before)

All under reconcile_from_streaming_config -> SketchStore::snapshot_instances(), which deep-clones every SketchInstanceMetadata (String + BTreeSet + AggKind strings) on each ingest batch:

• malloc 17.9% + cfree 5.7% (~23% allocation churn)
• BTreeMap::clone::clone_subtree 19.6% (children)
• SketchInstanceMetadata::clone 6.9% (children)
• String::clone 5.0%
• drop_in_place::<SketchInstanceMetadata> 2.1%
• reconcile_from_streaming_config itself 2.0% self / drives the above

What changed

1. Eliminate the per-reconcile catalog clone. New SketchStore::for_each_instance (lock-held visitor); reconcile now derives each sid's signature under the read lock into a reused scratch buffer, collects only u64 orphan sids, and retires them after the lock drops. No metadata clone, no per-sid signature Vec alloc.
2. Skip reconcile when config is unchanged. The streaming config is an Arc<ArcSwap<StreamingConfig>> whose Arc identity only changes on a (rare) control-plane swap. New reconcile_if_config_changed gates on the Arc data pointer via an AtomicUsize on SketchStore, collapsing the steady-state per-batch reconcile to one relaxed atomic load.

Measured after

Criterion reconcile_per_batch (full un-gated scan, isolating change #1):

sids	before	after	speedup
100	35.2 us	12.6 us	2.8x
1000	388 us	134 us	2.9x
10000	3.99 ms	1.45 ms	2.75x

Change #2 takes the steady-state per-batch cost to ~one atomic load (reconcile runs only on config swaps), removing this hotspot from the per-batch ingest path almost entirely.

Correctness

Reconcile only ever transitions Active->Retired on signature mismatch vs. the config (no time-based expiry — that stays in the eviction service), so an unchanged config yields identical results regardless of wall clock; a newly-minted sid matches a live signature by construction so it is never wrongly orphaned. Query answers are unaffected. All 9 reconcile lib tests pass (incl. 2 new gate tests + the HTTP config-swap path test); 228 sketch_db lib tests + edge-runtime adapter tests green.

Test plan

• cargo test -p data_plane --lib reconcile
• cargo test -p data_plane --lib sketch_db
• cargo bench -p data_plane --bench sketch_db -- reconcile_per_batch
• Re-profile live asap-backend to confirm the reconcile/clone frames are gone from the per-batch path.

Generated with Claude Code