SP-5/SP-8: EMA cost model, dynamic scraper, backend delivery - #86
Merged
Merged
Conversation
Adds a `query_parser` module that implements SP-1 workload extraction from raw query strings, covering DEBS 2022 financial queries and ClickBench SQL patterns, following the SQL-to-sketch mapping rules in the design doc. ## query_parser/promql.rs Regex-based PromQL parser recognising: quantile_over_time, avg/min/max/ sum/count_over_time, histogram_quantile, topk(k, count_over_time), and count(count_over_time ... by (dims)) for cardinality. Bare selectors (no agg fn) are marked exact_required for RSI/MACD/stochastic passthrough. ## query_parser/sql.rs sqlparser-AST traversal implementing the doc's Generate_SQL_Aggregation_ Sketch_Mapping rules: COUNT(*)+GROUP BY → Frequency; COUNT(DISTINCT) → Cardinality (HLL); AVG → Quantile p50; MIN/MAX → Quantile p0/p100; ORDER BY DESC LIMIT k → heavy-hitter CountSketch; SUM → exact. ## query_parser/mod.rs QueryHint enum for named DEBS patterns (DebsEma, DebsTopK, DebsPriceStats, DebsVolatility, DebsCardinality, DebsTwap, DebsAnomaly, ExactRequired). debs_hint() classifies financial.last_trade_price queries by quantile set. ## analyzer.rs Adds optional query_string field to QuerySpec. When provided, the parsed result populates metric_name, aggregations, time_window, group_by_labels, and label_filters; explicit fields override parsed values. ## types.rs / rules.rs Extends QueryWorkload with exact_required and quantiles fields. The RulesPlanner returns a raw-passthrough plan when exact_required=true and seeds DDSketch/KLL quantile params from parsed φ values. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…delivery (#80) - SP-5 (online_cost_model): expose OnlineMetricsStore from planner; wire into CostModelPlanner via with_online_store() so scoring uses EMA-blended costs - SP-8 (feedback loop): Scraper gains dynamic add/remove/set_sketch_type endpoints backed by Arc<RwLock<>>; new on_metrics callback feeds observed bandwidth + CPU/sample deltas into the EMA store after each scrape - OpAMP role tracking: X-Agent-Role header distinguishes agent vs backend collectors; push_to_role() delivers role-appropriate configs; on_connect / on_disconnect callbacks register/deregister scrape endpoints automatically - main.rs: initialise EMA store, wire Scraper + OpAMP callbacks, start scraper background loop, push agent config to Agent-role and backend config to Backend-role collectors on every plan Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…licts SP-1 query_parser (promql-parser 0.8, sqlparser 0.61) landed on main via PR #79 after controller-runtime branched. Take main's versions for Cargo.toml, Cargo.lock, and all query_parser files. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SieDeta
pushed a commit
that referenced
this pull request
Apr 17, 2026
* SP-1: PromQL and SQL query parsing with DEBS Q1–Q12 and sketch mapping Adds a `query_parser` module that implements SP-1 workload extraction from raw query strings, covering DEBS 2022 financial queries and ClickBench SQL patterns, following the SQL-to-sketch mapping rules in the design doc. ## query_parser/promql.rs Regex-based PromQL parser recognising: quantile_over_time, avg/min/max/ sum/count_over_time, histogram_quantile, topk(k, count_over_time), and count(count_over_time ... by (dims)) for cardinality. Bare selectors (no agg fn) are marked exact_required for RSI/MACD/stochastic passthrough. ## query_parser/sql.rs sqlparser-AST traversal implementing the doc's Generate_SQL_Aggregation_ Sketch_Mapping rules: COUNT(*)+GROUP BY → Frequency; COUNT(DISTINCT) → Cardinality (HLL); AVG → Quantile p50; MIN/MAX → Quantile p0/p100; ORDER BY DESC LIMIT k → heavy-hitter CountSketch; SUM → exact. ## query_parser/mod.rs QueryHint enum for named DEBS patterns (DebsEma, DebsTopK, DebsPriceStats, DebsVolatility, DebsCardinality, DebsTwap, DebsAnomaly, ExactRequired). debs_hint() classifies financial.last_trade_price queries by quantile set. ## analyzer.rs Adds optional query_string field to QuerySpec. When provided, the parsed result populates metric_name, aggregations, time_window, group_by_labels, and label_filters; explicit fields override parsed values. ## types.rs / rules.rs Extends QueryWorkload with exact_required and quantiles fields. The RulesPlanner returns a raw-passthrough plan when exact_required=true and seeds DDSketch/KLL quantile params from parsed φ values. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * SP-5/SP-8: runtime wiring — EMA cost model, dynamic scraper, backend delivery (#80) - SP-5 (online_cost_model): expose OnlineMetricsStore from planner; wire into CostModelPlanner via with_online_store() so scoring uses EMA-blended costs - SP-8 (feedback loop): Scraper gains dynamic add/remove/set_sketch_type endpoints backed by Arc<RwLock<>>; new on_metrics callback feeds observed bandwidth + CPU/sample deltas into the EMA store after each scrape - OpAMP role tracking: X-Agent-Role header distinguishes agent vs backend collectors; push_to_role() delivers role-appropriate configs; on_connect / on_disconnect callbacks register/deregister scrape endpoints automatically - main.rs: initialise EMA store, wire Scraper + OpAMP callbacks, start scraper background loop, push agent config to Agent-role and backend config to Backend-role collectors on every plan Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
online_cost_modelwired intoCostModelPlannerviawith_online_store(). Scoring now uses a live-blended cost table (70 % online / 30 % benchmark after 5+ observations) instead of static benchmarks, so the planner converges toward real-world behaviour over time.Scrapergains dynamic endpoint management (add_endpoint,remove_endpoint,set_sketch_type) backed byArc<RwLock<Vec<Endpoint>>>. Newon_metricscallback deliversScrapedData(sketch_size_bytes + derived cpu_micros_per_sample) to the EMA store after each successful scrape.X-Agent-Role: agent|backendheader distinguishes edge collectors from aggregation backends.push_to_role()delivers role-appropriate configs: agent YAML toAgentrole, backend YAML toBackendrole.on_connect/on_disconnecthooks auto-register/deregister scrape endpoints.Scraperwith EMA callback → buildOpampServerwith connect/disconnect hooks →CostModelPlanner::with_online_store→ start scraper background loop. After each plan: push agent config to agent-role collectors, backend config to backend-role collectors, update endpoint sketch types for EMA attribution.Test plan
cargo test— all 128 tests passmonitor— 8 tests including newon_metrics_callback_firesandadd_remove_endpointopamp— role parsing, push_to_role, connect/disconnectplanner::online_cost_model— EMA blending, store init, effective_table fallback🤖 Generated with Claude Code