refactor: SketchIndex on epoch-columnar storage (refactor #2)

[zzylol]

Summary

Refactor #2 in the chain after #120 (which scaffolded SketchIndex with naive HashMap<u64, Vec<SketchTimeSeries>> + BTreeMap<TimestampMs, …> per series). That naive shape collapsed all the optimizations the legacy SimpleMapStore had spent six attempts building. This PR lifts those primitives into a generic stores/epoch_columnar module and rebuilds SketchIndex.series on top of it.

No behavior change for upstream callers — runtime ingest/query paths still hit the legacy SimpleMapStore. The new types + tests + storage shape are in place, ready for the wire-in PR (Phase 4 + 5 + 6 finish).

Why

The legacy SimpleMapStore (asap-query-engine/src/stores/sketch_db/simple_map_store/) has an INDEX_DESIGN.md listing six storage optimizations:

Opt	What
1	Lazy window_to_ids — built on first exact query, invalidated cheaply on insert
2	Offset-based index — stores u32 column offsets, not payload clones
3	Monotonic ingest fast path — skip HashSet probe for consecutive same-window inserts
4	Batch metadata hoisting — caller groups DPs by key, fingerprints once
5	Columnar storage — three parallel arrays; range scan touches only windows_col
6	Pre-allocated epoch buffers on rotation

The new SketchIndex had none of them. Insert was Vec::push into per-sid Vec<SketchTimeSeries> with a linear scan to find the right series, then BTreeMap insert per sample. Range queries were O(per-sid Vec) × O(log N) per series. This PR carries Opts 1, 2, 3, 5, 6 into the new index; Opt 4 is caller-side and lands in the wire-in PR.

What changed

New: asap-query-engine/src/stores/epoch_columnar.rs (~470 lines)

Generic, parameterized over the payload type:

• InternTable<K: Eq + Hash + Clone> — interns a key (a per-series group-by VALUES vector for SketchIndex's use case) to LabelValuesId = u32. Hot path carries 4-byte ids instead of BTreeMap<String,String> clones.

• MutableEpoch<P> — append-only insert (amortized O(1)), three parallel arrays (windows / label-id / payload), monotonic same-window fast path (Opt 3), lazy window_to_ids (Opt 1+2, built on first exact_query, invalidated by single pointer-width write on insert), pre-allocated buffers on rotation (Opt 6).

• SealedEpoch<P> — from_mutable consumes the columns into a flat sorted Vec<(TimestampRange, LabelValuesId, P)> (O(M log M) paid once at rotation, off the insert hot path). Binary-search range scan O(log N + k). The consume avoids cloning payloads — they move out of the column.

• SidStoreData<K, P> — pairs one active MutableEpoch<P> with a BTreeMap<EpochId, SealedEpoch<P>> rotation ring. Rotation triggered by epoch_capacity distinct-window threshold; oldest sealed evicted at max_epochs.

Rewritten: asap-query-engine/src/stores/sketch_index.rs

type SidStore = Arc<RwLock<SidStoreData<BTreeMap<String, String>, SketchSampleState>>>;

pub struct SketchIndex {
    instances: RwLock<HashMap<u64, SketchInstanceMetadata>>,  // read-mostly
    series: DashMap<u64, SidStore>,                           // per-sid locking
}

• instances is RwLock<HashMap> because registration happens once per first-seen sid.
• series is DashMap because per-sid writes happen on every DP — one writer per sid does not block another.
• query_range(sid, t0, t1) is the new API replacing the missing range-query path: groups results back by interned label-values vector and returns Vec<SketchTimeSeries>.
• classify(sid) reads the per-sid SidStoreData to distinguish Hit (current_epoch or sealed_epochs nonempty) from Ghost (registered but empty) from Unknown (not registered).

Public surface preserved

• Capability, SketchKindHandle, SketchConfig, AccuracyBound,
  SketchInstanceMetadata, SketchSampleState, SketchEncoding,
  SidLookup — same shape, same pub use exports in stores/mod.rs.
• SketchTimeSeries is still returned by the query path; just no longer the storage shape.

Caveats

• The new SketchIndex is not yet on the runtime ingest or query path. Legacy SimpleMapStore (aggregation_id-keyed) still carries traffic. Wire-in lives in the next PR (Phase 4 + Phase 5 + Phase 6 finish).
• register and append_sample now take &self (interior mutability via RwLock + DashMap); call-sites that previously took &mut SketchIndex need a small adjustment.

Test plan

• cargo build --release -p query_engine_rust clean (warnings only, no errors)
• cargo build --release -p controller clean
• 10 unit tests pass (4 in epoch_columnar, 6 in sketch_index):
  • epoch_columnar::tests::intern_idempotent
  • epoch_columnar::tests::mutable_epoch_insert_and_query
  • epoch_columnar::tests::sealed_epoch_binary_search_range
  • epoch_columnar::tests::sid_store_rotation
  • sketch_index::tests::ghost_classification
  • sketch_index::tests::hit_after_append
  • sketch_index::tests::range_query_returns_distinct_series
  • sketch_index::tests::range_query_clips_to_window_bounds
  • sketch_index::tests::epoch_rotation_is_visible_to_query
  • sketch_index::tests::ddsketch_accuracy_bound
• Full cargo test --release -p query_engine_rust --lib stores — 182 passed; 0 failed (no regressions in legacy SimpleMapStore tests)
• Wire-in PR (Phase 4 + 5 + 6 finish): exercise the new index under live OTLP ingest + PromQL read

🤖 Generated with Claude Code