Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions crates/asap-aware-mapping/src/cost_model.rs
Original file line number Diff line number Diff line change
Expand Up @@ -440,7 +440,7 @@ pub trait CostModel {
/// how many structurally exist (`candidate.consumer_count`). See
/// `crate::recurrence`'s module docs for the full design.
///
/// - `recurrence.is_empty()` (no [`RepeatingEntry`]/[`DataCharacteristics`]-derived
/// - `recurrence.is_empty()` (no [`RepeatingEntry`]/[`DataWorkload`]-derived
/// metadata available): delegates to
/// [`cse_share_decision`](Self::cse_share_decision), preserving
/// today's structural-consumer-count behavior exactly — issue #287's
Expand All @@ -456,7 +456,7 @@ pub trait CostModel {
/// rates is equivalent to comparing `rate * H` for any fixed `H > 0`).
///
/// [`RepeatingEntry`]: asap_types::workload::RepeatingEntry
/// [`DataCharacteristics`]: asap_types::workload::DataCharacteristics
/// [`DataWorkload`]: asap_types::workload::DataWorkload
fn cse_share_decision_with_recurrence(
&self,
candidate: &CseCandidate,
Expand Down
6 changes: 3 additions & 3 deletions crates/asap-aware-mapping/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -204,9 +204,9 @@ pub use explanation::{
};
pub use grouping::{has_subpopulations, HydraGroupingStrategy};
pub use recurrence::{
evaluation_rate_of, total_cost, update_rate_from_data_characteristics, CostRate,
EvaluationRate, Horizon, RecurrenceCostExplanation, RecurrenceError, RecurrenceProfile,
RootRecurrence, UpdateRate,
evaluation_rate_of, total_cost, update_rate_from_data_workload, CostRate, EvaluationRate,
Horizon, RecurrenceCostExplanation, RecurrenceError, RecurrenceProfile, RootRecurrence,
UpdateRate,
};
pub use replacement::{
default_strategies, default_strategies_with, search_workload, search_workload_with,
Expand Down
99 changes: 49 additions & 50 deletions crates/asap-aware-mapping/src/recurrence.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
//!
//! ASAPPlanner already models recurring-workload metadata
//! ([`asap_types::workload::RepeatingEntry`]) and ingest-rate metadata
//! ([`asap_types::workload::DataCharacteristics`]), but until this module
//! ([`asap_types::workload::DataWorkload`]), but until this module
//! neither reached [`CostModel`]'s CSE share-vs-recompute decision
//! ([`CostModel::cse_share_decision`]): that decision only ever compared a
//! *structural* consumer count (how many workload locations reference a
Expand Down Expand Up @@ -79,12 +79,10 @@
//! for a whole workload). A one-shot ([`asap_types::workload::BatchEntry`])
//! consumer contributes to [`RecurrenceProfile::one_shot_consumers`]
//! instead, never to this rate.
//! - [`UpdateRate`]: derived from workload-level
//! [`asap_types::workload::DataCharacteristics`] via
//! [`update_rate_from_data_characteristics`] (`series_count *
//! samples_per_sec_per_series`) — a deployment with a more precise
//! per-target ingest measurement should compute its own `UpdateRate`
//! instead of relying on this proxy.
//! - [`UpdateRate`]: read from workload-level
//! [`asap_types::workload::DataWorkload::ingestion_rate`] via
//! [`update_rate_from_data_workload`]. Missing evidence remains unknown;
//! data at rest is not assigned a fabricated update rate.
//! - `maintenance_cost_per_update` / `summary_read_cost` /
//! `raw_recompute_cost`: [`CostModel`] hooks (defaults documented on the
//! trait itself, in `cost_model.rs`) — illustrative placeholders, like
Expand All @@ -95,7 +93,7 @@
//!
//! [`RecurrenceProfile::is_empty`] is `true` exactly when a caller supplied
//! no [`RepeatingEntry`](asap_types::workload::RepeatingEntry)/
//! [`DataCharacteristics`](asap_types::workload::DataCharacteristics)-derived
//! [`DataWorkload`](asap_types::workload::DataWorkload)-derived
//! information at all (no evaluation rate, no update rate, zero recorded
//! one-shot consumers — [`RecurrenceProfile::EMPTY`], its `Default`).
//! [`CostModel::cse_share_decision_with_recurrence`]'s default body checks
Expand All @@ -106,16 +104,16 @@

use std::fmt;

use asap_types::workload::{DataCharacteristics, RepetitionInterval};
use asap_types::workload::{DataWorkload, RepetitionInterval};

use crate::cost_model::{Cost, CostModel, CseCandidate, ShareDecision};

// ── Units ────────────────────────────────────────────────────────────────

/// How often the *raw* data underlying a maintained summary changes —
/// ingest/update events per second (Hz). See the module docs' provenance
/// table: normally derived from [`DataCharacteristics`] via
/// [`update_rate_from_data_characteristics`].
/// table: normally read from [`DataWorkload::ingestion_rate`] via
/// [`update_rate_from_data_workload`].
#[derive(Debug, Clone, Copy, PartialEq, PartialOrd)]
pub struct UpdateRate(pub f64);

Expand Down Expand Up @@ -234,7 +232,7 @@ pub enum RecurrenceError {
/// discipline [`evaluation_rate_of`] applies to each [`RepetitionInterval`],
/// applied at every point an `UpdateRate` enters a [`RecurrenceProfile`]
/// ([`RecurrenceProfile::with_update_rate`],
/// [`update_rate_from_data_characteristics`],
/// [`update_rate_from_data_workload`],
/// [`crate::replacement::PlanSpace::recurrence_profiles`]'s own parameter)
/// *and*, as a backstop that can't be bypassed by constructing a
/// `RecurrenceProfile` via its public fields directly, inside [`decide`]
Expand Down Expand Up @@ -284,24 +282,17 @@ where
Ok(any.then_some(EvaluationRate(total_hz)))
}

/// Derive an [`UpdateRate`] from workload-level [`DataCharacteristics`]:
/// `series_count * samples_per_sec_per_series` — the total number of raw
/// ingest samples per second across every series this characteristics
/// value describes. A proxy, not a measurement: a deployment with a more
/// precise per-target ingest rate should compute its own `UpdateRate`
/// rather than rely on this conversion.
///
/// Validated via [`validate_update_rate`]: `samples_per_sec_per_series` is
/// caller-supplied `f64` with no type-level guarantee of being finite or
/// non-negative, so a garbage `DataCharacteristics` value (NaN, infinite,
/// or negative) is rejected here rather than silently propagating into a
/// [`RecurrenceProfile`].
pub fn update_rate_from_data_characteristics(
dc: &DataCharacteristics,
) -> Result<UpdateRate, RecurrenceError> {
validate_update_rate(UpdateRate(
dc.series_count as f64 * dc.samples_per_sec_per_series,
))
/// Read an [`UpdateRate`] from workload-level [`DataWorkload`] evidence.
/// Missing evidence remains `None`; a present non-finite or negative rate is
/// rejected rather than propagated into a [`RecurrenceProfile`].
pub fn update_rate_from_data_workload(
workload: &DataWorkload,
) -> Result<Option<UpdateRate>, RecurrenceError> {
workload
.ingestion_rate
.value
.map(|rate| validate_update_rate(UpdateRate(rate.0)))
.transpose()
}

// ── RecurrenceProfile ────────────────────────────────────────────────────
Expand All @@ -326,7 +317,7 @@ pub struct RecurrenceProfile {
pub one_shot_consumers: usize,
/// The ingest/update rate of the raw data this target (if maintained)
/// would be kept up to date against. `None` when no
/// [`DataCharacteristics`] were available.
/// [`DataWorkload::ingestion_rate`] evidence was available.
pub update_rate: Option<UpdateRate>,
}

Expand Down Expand Up @@ -365,7 +356,7 @@ impl RecurrenceProfile {
self
}

/// Attach an ingest/update rate (from [`update_rate_from_data_characteristics`]
/// Attach an ingest/update rate (from [`update_rate_from_data_workload`]
/// or a deployment-specific measurement). Validated via
/// [`validate_update_rate`] — rejects a NaN, infinite, or negative rate
/// rather than silently storing it.
Expand Down Expand Up @@ -670,34 +661,42 @@ mod tests {
assert_eq!(err, RecurrenceError::InvalidInterval(interval(0)));
}

// ── update_rate_from_data_characteristics ────────────────────────────
// ── update_rate_from_data_workload ───────────────────────────────────

#[test]
fn update_rate_from_data_characteristics_multiplies_series_by_sample_rate() {
let dc = DataCharacteristics {
series_count: 1_000,
samples_per_sec_per_series: 0.1,
bytes_per_raw_sample: 80,
distinct_keys_per_window: None,
data_distribution: Default::default(),
fn update_rate_from_data_workload_reads_ingestion_rate_evidence() {
let workload = DataWorkload {
ingestion_rate: asap_types::workload::Evidence {
value: Some(asap_types::workload::Rate(100.0)),
..Default::default()
},
..Default::default()
};
let rate = update_rate_from_data_characteristics(&dc).unwrap();
let rate = update_rate_from_data_workload(&workload).unwrap().unwrap();
assert!((rate.0 - 100.0).abs() < 1e-9);
}

#[test]
fn update_rate_from_data_characteristics_rejects_a_negative_sample_rate() {
let dc = DataCharacteristics {
series_count: 1_000,
samples_per_sec_per_series: -0.1,
bytes_per_raw_sample: 80,
distinct_keys_per_window: None,
data_distribution: Default::default(),
fn update_rate_from_data_workload_rejects_a_negative_rate() {
let workload = DataWorkload {
ingestion_rate: asap_types::workload::Evidence {
value: Some(asap_types::workload::Rate(-0.1)),
..Default::default()
},
..Default::default()
};
let err = update_rate_from_data_characteristics(&dc).unwrap_err();
let err = update_rate_from_data_workload(&workload).unwrap_err();
assert!(matches!(err, RecurrenceError::InvalidUpdateRate(_)));
}

#[test]
fn update_rate_from_data_workload_preserves_missing_evidence() {
assert_eq!(
update_rate_from_data_workload(&DataWorkload::default()).unwrap(),
None
);
}

// ── RecurrenceProfile ─────────────────────────────────────────────────

#[test]
Expand Down Expand Up @@ -1074,7 +1073,7 @@ mod tests {
assert_eq!(explanation.recompute_total, Some(Cost(50.0)));
}

/// A batch-only workload (no `DataCharacteristics`, only one-shot
/// A batch-only workload (no `DataWorkload`, only one-shot
/// consumers) with the *default* `DefaultCostModel` must not
/// unconditionally prefer `Share` regardless of how many one-shot
/// consumers there are — issue #287 review bug 1's original repro,
Expand Down
6 changes: 3 additions & 3 deletions crates/asap-aware-mapping/src/replacement.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2311,7 +2311,7 @@ impl<Id> PlanSpace<Id> {

/// One [`RecurrenceProfile`] per discovered [`MemoGroup`] target, built by
/// [`PlanSpace::recurrence_profiles`] — the "carry `RepeatingEntry.interval`
/// and relevant `DataCharacteristics` into ASAP-aware search/cost context"
/// and relevant `DataWorkload` into ASAP-aware search/cost context"
/// half of issue #287. Looked up by `Rc` pointer identity, the same
/// currency [`PlanSpace::group_for`]/[`GlobalSelection::for_target`] already
/// use.
Expand Down Expand Up @@ -2372,10 +2372,10 @@ impl<Id> PlanSpace<Id> {
/// `update_rate` is applied uniformly to every discovered site *that
/// this walk actually reached from some root* (see the "unreachable
/// sites" note below): today's
/// [`asap_types::workload::DataCharacteristics`] is a single
/// [`asap_types::workload::DataWorkload`] is a single
/// workload-level value (applies to every query in a `QueryWorkload`),
/// not per-target, so there is no finer-grained source to attach
/// instead. `None` when no `DataCharacteristics` were available —
/// instead. `None` when no `DataWorkload` evidence was available —
/// preserves "missing metadata" behavior for the update-rate term alone
/// even when repeating/one-shot consumer information is present.
///
Expand Down
4 changes: 2 additions & 2 deletions crates/frontend-promql/tests/promql_lowering.rs
Original file line number Diff line number Diff line change
Expand Up @@ -769,7 +769,7 @@ fn batch_lowers_each_entry_and_reads_per_query_accuracy() {
},
]),
repeating_queries: None,
data_characteristics: None,
data_workload: None,
};
let results = lower_promql_batch(&workload);
assert_eq!(results.len(), 2);
Expand All @@ -787,7 +787,7 @@ fn batch_rejects_non_promql_language() {
requirements: None,
}]),
repeating_queries: None,
data_characteristics: None,
data_workload: None,
};
let results = lower_promql_batch(&workload);
assert_eq!(results.len(), 1);
Expand Down
87 changes: 63 additions & 24 deletions crates/types/src/workload.rs
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ pub struct BatchEntry {
pub requirements: Option<QueryRequirements>,
}

/// One entry in a repeating (streaming) workload: a query that fires
/// every `interval` milliseconds.
/// One query that fires every `interval` milliseconds. Its recurrence does
/// not imply that the queried data is continuously ingesting.
#[derive(Debug, Clone)]
pub struct RepeatingEntry {
pub query: Query,
Expand All @@ -60,7 +60,18 @@ pub struct RepeatingEntry {
pub requirements: Option<QueryRequirements>,
}

// ── Data characteristics ──────────────────────────────────────────────────────
// ── Data workload ─────────────────────────────────────────────────────────────

/// Whether the data queried by this workload is static, still arriving, or a
/// mixture of both. This is independent of whether queries repeat.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum DataArrival {
AtRest,
ContinuouslyIngesting,
Mixed,
#[default]
Unknown,
}

/// Statistical distribution of keys in the incoming data stream.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
Expand All @@ -78,23 +89,51 @@ pub enum DataDistribution {
Bursty,
}

/// Characteristics of the data arriving at the ingestion layer.
/// Used by the cost model and sketch-parameter binder to size sketches
/// and estimate transmission cost without running the query.
#[derive(Debug, Clone)]
pub struct DataCharacteristics {
/// Number of distinct active time series for this metric.
pub series_count: u64,
/// Sample rate per series at the SDK / agent (Hz).
pub samples_per_sec_per_series: f64,
/// Wire size of one raw OTLP metric data point after protobuf encoding
/// (bytes). Typical range: 50–200 bytes.
pub bytes_per_raw_sample: u32,
/// Known distinct key values per flush period for frequency / cardinality
/// sketches. `None` → inferred analytically from inserts and distribution.
pub distinct_keys_per_window: Option<u64>,
/// Statistical distribution of keys in the stream.
pub data_distribution: DataDistribution,
/// Where an empirical workload value came from.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum EvidenceSource {
Declared,
Observed,
Derived,
#[default]
Unknown,
}

/// A workload value together with the provenance and freshness needed to
/// decide whether it is safe to use. Times and durations are milliseconds.
#[derive(Debug, Clone, PartialEq)]
pub struct Evidence<T> {
pub value: Option<T>,
pub source: EvidenceSource,
pub observed_at_ms: Option<u64>,
pub valid_for_ms: Option<u64>,
}

impl<T> Default for Evidence<T> {
fn default() -> Self {
Self {
value: None,
source: EvidenceSource::Unknown,
observed_at_ms: None,
valid_for_ms: None,
}
}
}

/// Queries per second, samples per second, or another rate whose unit is
/// established by the field that contains it.
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct Rate(pub f64);

/// Workload-level facts about the data being queried. Unlike the former
/// ingestion-only `DataCharacteristics`, this also represents data at rest.
#[derive(Debug, Clone, Default, PartialEq)]
pub struct DataWorkload {
pub arrival: DataArrival,
pub ingestion_volume: Evidence<u64>,
pub ingestion_rate: Evidence<Rate>,
pub input_cardinality: Evidence<u64>,
pub distribution: Evidence<DataDistribution>,
}

// ── Top-level workload ────────────────────────────────────────────────────────
Expand All @@ -111,9 +150,9 @@ pub struct QueryWorkload {
pub language: QueryLanguage,
/// One-shot queries executed together as a batch.
pub query_batch: Option<Vec<BatchEntry>>,
/// Queries that repeat on a fixed interval (streaming / continuous).
/// Queries that repeat on a fixed interval.
pub repeating_queries: Option<Vec<RepeatingEntry>>,
/// Workload-level data characteristics used for sketch sizing and cost
/// estimation. Applies to all queries in this workload.
pub data_characteristics: Option<DataCharacteristics>,
/// Workload-level data facts used for accuracy and cost estimation.
/// Applies to all queries in this workload.
pub data_workload: Option<DataWorkload>,
}
3 changes: 3 additions & 0 deletions docs/design_docs/asap-aware-mapping/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,9 @@ The design is split into focused documents:
summaries and optimizations can be composed safely.
- [End-to-end accuracy guarantees](end-to-end-accuracy-guarantees.md) specifies the typed
guarantee IR, sketch contracts, composition rules, target checking, and fail-closed boundaries.
- [Query workloads, data workloads, and summary lifecycle maintenance](workload-demand-and-summary-lifecycle.md) separates
query-workload properties from data-workload properties and defines ephemeral, prepared,
shared, and continuously maintained summary-state alternatives.
- [Explainability](explainability.md) describes how the planner reports available replacements
using the same candidate space it optimizes.

Expand Down
Loading
Loading