feat(sid): add find_matching_policies for Candidate → policy_fp lookup - #204
Merged
Merged
Conversation
…p lookup Closes the analyzer-side half of the merged-sid-identity query-path index. PR #203 added `SketchStore::sids_for_policy(fp) → [sid]`; this PR adds `find_matching_policies(registry, candidate) → [fp]` so the query engine can do `Candidate → [fp] → [sid]` in two O(1)-amortized hops instead of walking the sid metadata map. ## What - New `control_plane::warm_tier_analysis::policy_capability(cfg)` — maps `AggregationConfig` to the warm-tier `Capability` its sids serve. Inverse of `capability_for(&AggIntent)`: where the latter says "this intent wants *this* capability", this says "this stored policy *provides* this capability". - New `control_plane::warm_tier_analysis::find_matching_policies(registry, candidate)` — walks the policy registry, returns every fingerprint whose policy satisfies the candidate. Predicate (all must hold): 1. `policy.metric == candidate.metric_name` 2. `candidate.group_by_keys ⊆ policy.grouping_labels` (extra policy keys are fine; query can re-aggregate down) 3. `policy_capability(policy)` exists and is satisfied by `candidate.required_capability` (uses the existing `Capability::is_satisfied_by` semantics) 4. `policy.window_size ≤ candidate.range_seconds` (finer windows answer coarser queries via merge; reverse isn't true). `range_seconds == 0` (instant-vector) bypasses this check. 5. `policy.spatial_filter_normalized.is_empty()` — only unfiltered policies for now (candidate doesn't carry a filter shape; filtered match is a future enhancement) - New dep: `control_plane → asap_types` (`PolicyRegistry`, `PolicyFingerprint`, `AggregationConfig`). Added to `Cargo.toml`. ## Mapping table — `policy_capability(cfg)` | `AggregationType` | `Capability` | |---|---| | `Sum` | `ExactAgg(Sum)` | | `Increase` | `ExactAgg(Increase)` | | `MinMax` | `ExactAgg(MinMax)` | | `DDSketch` | `QuantileApprox(DDSketch)` | | `DatasketchesKLL` | `QuantileApprox(Kll)` | | `HLL` | `CardinalityApprox` | | `CountMinSketch` | `FrequencyEstimate(CountMin)` | | `CountSketch` | `FrequencyEstimate(CountSketch)` | | `CountMinSketchWithHeap` | `FrequencyTopk(CmsWithHeap)` | | multi-pop variants (`Multiple*`, `HydraKLL`), legacy wrappers, set-aggregators | `None` (skipped) | The multi-pop variants stay `None` until the keyed-ExactAgg follow-up lands an L4 binder for them. Adding a `Capability::ExactAgg(MultipleSum)` arm here today would surface candidates the planner can't route. ## End state of the query-path index ``` PromQL → analyzer → WarmTierCandidate │ ▼ find_matching_policies(registry, &candidate) │ ← O(N_policies); typically ~thousands, not millions ▼ Vec<PolicyFingerprint> │ ▼ SketchStore::sids_for_policy(fp) │ ← O(1) per fp; PR #203 ▼ Vec<sid> │ ▼ SketchStore::range_query + SketchReducer::evaluate ``` ## What's not in scope - Candidate-side spatial-filter shape. Today filtered policies are unconditionally skipped; the candidate has no filter to compare against. When the analyzer surfaces a filter, this predicate gains a `candidate.spatial_filter_normalized == policy.spatial_filter_normalized` arm. - Wire-up to the query engine. `ASAPQueryEngine.execute` still walks `instances_matching(metric, gbk)` today; this PR is the lookup primitive, not the integration. A follow-up swaps the call. ## Test plan - [x] 14 new unit tests covering each match / mismatch dimension - [x] `cargo check --workspace` clean - [x] `cargo test -p control_plane --lib warm_tier_analysis::tests::matching` — 15 passed - [x] `cargo test --workspace --lib --bins` green except for the pre-existing `avg_finds_sum_and_count` HashMap-iteration flake 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
added a commit
that referenced
this pull request
May 14, 2026
…205) Wires up the OTel sketch-ingest path to populate `SketchInstanceMetadata.policy_fp` instead of leaving it `UNSET`. Sketch-backed sids now participate in the `policy_fp → {sids}` reverse index (PR #203), so the analyzer's `find_matching_policies` (PR #204) returns fingerprints whose sids are O(1)-reachable. ## What - New `control_plane::warm_tier_analysis::find_policy_by_content( registry, metric, group_by_keys, agg_type, expected_params)` — finds the policy whose contents match a freshly-ingested sketch's shape. Returns `Some(fp)` on a unique match, `None` on zero or multiple matches (ambiguous → stay UNSET). - Two data-plane helpers in `data_plane/src/drivers/ingest/otel.rs`: - `aggregation_type_for_sketch_handle(SketchKindHandle) → Option<AggregationType>` - `sketch_config_to_params(&SketchConfig) → HashMap<String, Value>` Both lock in the data-plane → control-plane wire-shape mapping so drift surfaces as test failures, not silent lookup misses. - New `derive_sketch_policy_fp(ingest_state, metric, kind, cfg, group_by_keys)` threads the content match. Called from the OTel sketch ingest registration site (`route_modified_otlp_sketches_to_precompute`). Returns `PolicyFingerprint::UNSET` when no policy matches — sids stay reachable via the legacy `instances_matching(metric, gbk)` walk. ## Matching shape Match requires all of: 1. `policy.metric == metric` 2. `policy.aggregation_type == agg_type` (mapped from `SketchKindHandle`) 3. `policy.grouping_labels` (as a set) == `group_by_keys` 4. For every key in `expected_params`, `policy.parameters` has the same `serde_json::Value` (extra policy params tolerated) 5. `policy.spatial_filter_normalized.is_empty()` — OTLP sketches don't carry a filter context Ambiguous match (multiple policies → same shape) returns `None` intentionally. Such policies would have collided on sid identity anyway — surfacing as UNSET is the honest signal of a control-plane bug. ## Param-key vocabulary | `SketchConfig` | params key(s) | |---|---| | `DDSketch { relative_accuracy }` | `relative_accuracy` | | `Kll { k }` | `k` | | `Hll { precision }` | `precision` | | `CountSketch { rows, cols }` | `rows`, `cols` | | `CountMin { rows, cols }` | `rows`, `cols` | Names must stay in sync with `AggregationConfig::from_yaml_data` in `asap_types/src/aggregation_config.rs`. The new test `sketch_config_to_params_uses_canonical_keys` locks them. ## What's still standing - OTel sketch-ingest doesn't surface a spatial-filter context, so filtered policies remain unreachable from this path. When the agent's processor emits a filter shape in the OTLP DP, plumb it through to the lookup. - Keyed-group ExactAgg (`MultipleSum` / `MultipleIncrease`) policies don't appear in `policy_capability` either; if/when those land an L4 binder, both this lookup and `find_matching_policies` need matching arms. ## Test plan - [x] 2 new unit tests in `otel::policy_fp_lookup_tests` covering the `SketchKindHandle → AggregationType` mapping and the `SketchConfig → params` rendering - [x] `cargo check --workspace` clean - [x] `cargo test --workspace --lib --bins` green (pre-existing `avg_finds_sum_and_count` HashMap-iteration flake unchanged) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2 tasks
zzylol
added a commit
that referenced
this pull request
May 14, 2026
…ryEngine.execute (#206) Switches the query engine's candidate → sid resolution to the content-addressed fast path. PRs #203, #204, #205 landed the primitives; this PR integrates them. ## Before vs. after Before, per candidate: 1. `idx.instances_matching(metric, gbk)` walks the `RwLock<HashMap<u64, SketchInstanceMetadata>>` and returns every sid whose metric + group-by-keys match. 2. Per-sid: classify, check capability satisfaction, push to `hit_sids`. After, per candidate: 1. Snapshot the streaming config; build the `PolicyRegistry`. 2. `find_matching_policies(registry, candidate)` — walks the small policy registry (≤ thousands of entries) with the full match predicate (metric + group_by + capability + window + filter). Returns `Vec<PolicyFingerprint>`. 3. For each fp: `idx.sids_for_policy(fp)` — O(1) hash lookup over the reverse index added in PR #203. 4. Same per-sid classify + capability check as before (defensive; fast-path-discovered sids already satisfy the candidate by construction, but UNSET sids reached via the fallback don't). ## Slow-path fallback retained When the fast path yields zero sids — because either: - no policy in the registry matches the candidate (control plane hasn't published one yet), OR - the candidate's sids were registered with `PolicyFingerprint::UNSET` (legacy paths that didn't carry an `AggregationConfig` at ingest, test fixtures, raw mode) — the engine falls back to the metadata walk `instances_matching(metric, gbk)`. The per-sid capability filter below catches mismatches the fast path would have rejected at policy-match time. The fallback can be deleted in a follow-up once every code path populates `policy_fp` and existing on-disk records have aged out. ## Snapshot semantics The streaming-config snapshot is pinned once per query (not per candidate). Hot-reload swaps the underlying `Arc<StreamingConfig>` mid-query are isolated by the snapshot: the query sees the policy set that was active at query start. Same isolation the legacy `streaming_config_snapshot()` call already gave the engine elsewhere. ## Test plan - [x] `cargo check --workspace` clean - [x] `cargo test --workspace --lib --bins` green (`asap_types::capability_matching::tests::avg_finds_sum_and_count` HashMap-iteration flake unchanged) - Existing query-engine tests cover the slow-path fallback (their fixtures register sids with UNSET fp); future tests on the fast path land alongside an end-to-end test that builds a streaming config with matching policies and verifies the fp-keyed lookup is used. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the analyzer-side half of the merged-sid-identity query-path index. PR #203 added
SketchStore::sids_for_policy(fp) → [sid]; this PR addsfind_matching_policies(registry, candidate) → [fp].What
policy_capability(cfg: &AggregationConfig) -> Option<Capability>— inverse ofcapability_for(&AggIntent). Maps eachAggregationTypeto the warm-tierCapabilityits sids serve.find_matching_policies(registry, candidate) -> Vec<PolicyFingerprint>— walks the policy registry; returns every fingerprint whose policy satisfies the candidate.Matching predicate
policy.metric == candidate.metric_namecandidate.group_by_keys ⊆ policy.grouping_labelspolicy_capability(policy)isSome(c)andcandidate.required_capability.is_satisfied_by(&c)policy.window_size ≤ candidate.range_seconds(instant-vector queries withrange_seconds=0bypass)policy.spatial_filter_normalized.is_empty()— only unfiltered policies for nowNot in scope
WarmTierCandidate— filtered policies are unconditionally skipped today.ASAPQueryEngine.execute— this is the primitive, not the integration.Test plan
warm_tier_analysis::tests::matchingcargo check --workspacecleancargo test --workspace --lib --binsgreen (pre-existing HashMap-iteration flake unchanged)🤖 Generated with Claude Code