Skip to content

feat(emit): inject attributes/keep upstream of each sketch processor (B3) - #278

Merged
zzylol merged 1 commit into
mainfrom
b3-attributes-keep-grouping-labels
May 18, 2026
Merged

feat(emit): inject attributes/keep upstream of each sketch processor (B3)#278
zzylol merged 1 commit into
mainfrom
b3-attributes-keep-grouping-labels

Conversation

@zzylol

@zzylol zzylol commented May 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a transform/keep_for_<sanitized_metric> OTTL processor in front of every sketch processor the controller emits, so the agent reduces wire attrs to streaming_config.grouping_labels before sketching.
  • New EdgeStageConfig::metric_to_grouping_labels field carries the per-metric allowlist; populated from QueryWorkload::group_by_labels by the main bootstrap, typed-stage-split, and replanner paths.
  • Both emit paths covered: the 5-sketch routing-connector path (per-family pipelines) and the legacy single-pipeline path.

Why

Today the controller emits sketch processors with no preceding attribute-reduction step. Real producers emit metrics with a wide attribute set (e.g. {zone, rack, node, pod, endpoint, service.name, telemetry.sdk.*} = 9 keys), so the sketch processor mints one sid per unique attr tuple — 51 sids for http_requests_total_latency_ms in the smoke test. With B3, sids are keyed on the streaming-config's grouping_labels (e.g. one per zone).

Pairs with PR #273 (sid-resolution side fix): #273 makes the query path find UNREDUCED sids via subset-matching; this PR removes the need for the subset walk by ensuring the agent emits REDUCED sids in the first place. Both stay landed — the subset walk is the fallback for non-controller-managed sketches.

Implementation notes

  • OTTL transform over attributes/keep: the attributes processor has no native allowlist action (only insert/update/delete/hash). OTTL's keep_keys(datapoint.attributes, [...]) is the right primitive and the transform processor is already in the asap-otel builder-config alongside attributes, filter, and groupbyattrs.
  • Naming: transform/keep_for_<sanitized_metric> where the sanitiser replaces every non-[A-Za-z0-9_] byte with _ (OTel component-id grammar rejects dots/dashes/slashes).
  • where metric.name == "<metric>" guard: makes each statement a no-op on non-matching metrics — important because per-family pipelines see ALL metrics routed to that family by the connector, not just the controller's currently-planned one.
  • Pipeline ordering: [memory_limiter, gorillas3?, transform/keep_for_<metric>*, <family>processor, batch]. gorillas3 writes the cold tier on RAW samples (full wire attrs preserved in MinIO for drill-down) BEFORE the keep-processor strips attrs for the sketch's benefit.
  • Empty grouping_labels supported — emits keep_keys(datapoint.attributes, []) (planner's signal for one global sid per metric).
  • Backward-compat: empty map → no transform/keep blocks emitted (raw OTel agents that bypass the typed-stage-split keep their pre-B3 wire shape; attrs flow through unmodified).

Test plan

  • cargo test -p control_plane --lib — 699/699 pass (691 baseline + 8 new B3 tests).
  • cargo build -p control_plane --lib — clean (only pre-existing warnings).
  • End-to-end smoke test: expect schema endpoint per-metric sid count for http_requests_total_latency_ms to drop from 51 to the cardinality of unique zone values.

New tests in control_plane/src/emit/stage_config.rs::tests:

  • b3_emits_transform_keep_processor_per_metric_with_grouping_labels
  • b3_transform_block_uses_keep_keys_ottl_with_correct_labels
  • b3_per_family_pipeline_prepends_keep_before_sketch
  • b3_keep_lives_after_gorillas3_so_cold_tier_keeps_full_attrs
  • b3_processor_name_sanitises_metric_special_chars
  • b3_no_transform_processor_when_grouping_labels_absent
  • b3_empty_grouping_label_list_emits_empty_keep_keys
  • b3_legacy_emit_edge_yaml_injects_keep_for_source_metric

🤖 Generated with Claude Code

@zzylol @claude
refactor(query): retire legacy handle_query path + delete ast_matchin…
40e6a85 
…g crate (B7.5, schema-retirement #5 step 4)

After #273 (modern execute() handles sketches via instances_matching)
and #277 (saturating_sub for time=0 + dangling-callers cleanup), the
legacy `handle_query` / `handle_query_promql` family no longer has
features the modern `QueryEngine::execute()` trait can't handle. This
PR deletes the whole legacy tree and the orphan
`crates/promql_utilities/src/ast_matching/` module that only the
legacy path consumed.

Deleted:
  * `ASAPQueryEngine::handle_query`,
    `ASAPQueryEngine::handle_query_promql`,
    `ASAPQueryEngine::try_handle_query_promql_via_timeline`
    (instant-query entry points)
  * `ASAPQueryEngine::handle_range_query_promql`,
    `build_range_query_execution_context_promql`,
    `execute_range_query_pipeline`,
    `handle_binary_expr_range_promql`,
    `build_arm_range_context`, `apply_range_binary_op`
    (range-query entry point + helpers)
  * `execute_context`, `execute_query_pipeline`,
    `execute_store_query`, `execute_and_merge_store_queries`
    (legacy dispatcher chain)
  * `build_query_execution_context_promql*`,
    `build_promql_execution_context_tail`,
    `parse_and_match_promql`, `resolve_agg_info_promql`,
    `agg_info_from_forced_id`,
    `find_compatible_aggregation_with_miss_notify`,
    `resolve_metric_labels`,
    `calculate_query_timestamps_promql`,
    `calculate_start_timestamp_promql`,
    `validate_and_align_end_timestamp`,
    `extract_quantile_param_promql`, `extract_topk_param`,
    `build_query_kwargs_promql`, `create_keys_query_params`,
    `create_store_query_plan`, `collect_all_results`,
    `merge_precomputed_outputs`, `merge_accumulators`,
    `collect_results_separate_keys`,
    `collect_results_same_aggregation`, `limit_keys_for_topk`,
    `validate_range_query_params`, `format_final_results`,
    `build_query_requirements_promql`,
    `query_precompute_for_statistic` (legacy helpers)
  * `QueryExecutionContext`, `QueryMetadata`, `QueryTimestamps`,
    `StoreQueryParams`, `StoreQueryPlan`, `RangeQueryParams`,
    `RangeQueryExecutionContext` (legacy types)
  * `control_plane_patterns` field, the `PromQLPatternBuilder`
    setup in `new_with_hot_reload`, `QueryPatternType` enum
  * `crates/promql_utilities/src/ast_matching/` (4 files; only
    consumer was the legacy path)
  * `crates/promql_utilities/src/query_logics/parsing.rs` helpers
    (`get_metric_and_spatial_filter`, `get_statistics_to_compute`,
    `get_spatial_aggregation_output_labels`)
  * Test modules tied to the deleted surface: `range_query_tests`,
    `sketch_query_tests`, `e2e_feedback_loop_tests`,
    `forced_agg_id_tests`, `hll_count_query_tests`,
    `kll_quantile_query_tests`, `cms_rate_capability_tests`,
    `analyzer_parity_tests`,
    `calculate_start_timestamp_promql_tests`,
    `aux_pushdown_tests`, the whole
    `capability_matching_tests.rs` file and
    `tests/test_utilities/comparison.rs`

Updated:
  * `data_plane/src/drivers/query/servers/http.rs::process_via_simple_engine`
    now calls modern `execute()` only. The capability-miss notify
    side-effect that used to live in
    `find_compatible_aggregation_with_miss_notify` is moved to
    the modern path's sid-resolution error branches AND to the
    "no-sketch-index attached" branch (HttpServer attaches its own
    `SketchStore` but the `ASAPQueryEngine` builder it hands off
    does not `.with_sketch_index(...)` — the e2e test
    `http_capability_miss_feedback_loop_closes_over_http` pins
    exactly that wiring).
  * `handle_range_query` now calls modern
    `execute_range_promql_modern` only — no legacy fallback.
  * `handle_precompute_job` routes through modern `execute()`.
  * `tests/schema_timeline_dispatch_tests.rs::single_schema_query_falls_through_to_default_path`
    moved to `#[ignore]` — its premise no longer has a callsite.
    Modern-path coverage lives in `asap_tier_classify_tests` and
    `e2e_modified_otlp_sketch_path`.

Test plan:
  * `cargo test -p data_plane --lib`: 707 pass, 0 failed, 5 ignored
    (down from 754 — 47 tests deleted with the legacy code they
    exercised).
  * `cargo test -p data_plane --lib capability_miss_http_e2e`:
    1 pass, 1 ignored — the feedback-loop test that pins the
    notify side-effect still passes.
  * `cargo test -p control_plane --lib`: 699 pass, 0 failed.

The `try_handle_query_promql_via_timeline` cross-reconfigure
dispatch path is gone too. Its functionality (per-segment dispatch
across schema boundaries) was scheduled for a sid-level rewrite
in the schema-retirement #5 follow-up; deferred to a separate PR
since no current test exercises a multi-segment reconfigure
boundary (the two `#[ignore]`d tests in
`tests/schema_timeline_dispatch_tests.rs` documented as needing a
sid-level rewrite anyway).

Closes step 4 of #272.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit e8c205e into main May 18, 2026
This was referenced May 18, 2026
Merged
Closed
@zzylol
zzylol deleted the b3-attributes-keep-grouping-labels branch July 17, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zzylol