mvp phase ε.1: three-mode cost model with native OTLP to Prometheus (Mode 3) - #319
Merged
Merged
Conversation
Phase ε.1 collapses the planner's raw-vs-sketch + edge-vs-backend axes into a single tri-mode selector. Adds: - `planner::wire_cost` — per-sketch DELTA-encoded wire-cost table, break-even-samples math, and `BindMode` selection rules. Resulting break-evens at 50 B per raw sample: DDSketch+delta=16, KLL=64, HLL+delta=204, Count-Min+delta=84, Count-Sketch+delta=5,004. - `SketchExpr::RawAtEdgeSketchAtBackend` (Mode 2) and `SketchExpr::RawAtEdgePrometheusArchive` (Mode 3) variants. - Allocator coloring + emitter walk for both new variants. - `BackendAggregation::aggregation_input` (sketch_envelope vs raw) and `EdgeStageConfig::prometheus_archive_metrics` carried through. KLL has no delta variant per Implementation.tex — it ships full state; documented in the cost-table doc string. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the L5 emitters for the three-mode placements:
- `emit_edge_yaml` adds an `otlphttp/prometheus` exporter pointing at
Prometheus's native OTLP receiver (`/api/v1/otlp/v1/metrics`) when
`prometheus_archive_metrics` is non-empty. A `routing` processor
dispatches per-metric on `attributes["asap.mode"]` between the
warm-tier pipeline (sketch processors → otlp/backend) and the
archive pipeline (passthrough → otlphttp/prometheus).
- `emit_backend_config_json` surfaces `aggregationInput` so the
backend's StreamingConfig consumer knows whether to expect sketch
envelopes (Mode 1) or raw OTLP samples to sketch at ingest (Mode 2).
- `emit_backend_storage_routing_with_prometheus` adds Mode-3 metrics
with a single `engine: prometheus_remote` target — Phase ε.2
registers the engine on the backend HTTP query path.
Edge-side YAML uses OTLP everywhere on the wire (no
prometheusremotewrite). Endpoint is `${ASAP_PROMETHEUS_OTLP_URL}`-
overridable for the deploy team.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the deploy-side artefacts for Mode 3 (RawAtEdgePrometheusArchive): - `deploy/configs/prometheus-otlp-receiver.yml` — Prometheus 2.47+ config for native OTLP ingest. Sets `out_of_order_time_window: 30m` to match the existing Thanos overlay's tolerance, and `promote_resource_attributes` for the standard OTel resource fields the `prometheus_remote` engine's PromQL queries rely on. - `deploy/docker-compose/mvp-prometheus-archive.yml` — overlay that brings up Prometheus with `--web.enable-otlp-receiver` AND `--enable-feature=otlp-deltatocumulative`. The latter is required so agents don't need a separate cumulativetodelta processor before exporting OTel-SDK delta-encoded counters. Step-2.4 demo runs that want to exercise Mode 3 should layer this overlay on top of `mvp-multi-stage.yml`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase ε.1 collapses the planner's raw-vs-sketch + edge-vs-backend axes
into a single tri-mode
BindModeselector. All three modes use OTLPon the wire — Mode 3 ships raw OTLP directly to Prometheus's native
OTLP receiver at
/api/v1/otlp/v1/metrics(Prom 2.47+, activated with--web.enable-otlp-receiver).Three modes:
(default for high samples_per_window).
builds sketch at ingest (when
edge_cpu_budgetis tight). Phase ε.2implements the raw-input ingest path.
Prometheus; backend HTTP-forwards queries to
/api/v1/query. Phaseε.2 wires the
prometheus_remoteengine on the backend.Changes
planner::wire_cost— DELTA-encoded per-sketch wire-cost table,break-even-samples math, three-mode
BindModewith the Phase-ε.1selection rules.
SketchExpr::RawAtEdgeSketchAtBackendandSketchExpr::RawAtEdgePrometheusArchivevariants + allocator /emitter walks.
BackendAggregation::aggregation_input(sketch_envelope vs raw) andEdgeStageConfig::prometheus_archive_metrics.emit_edge_yamladds anotlphttp/prometheusexporter +routingprocessor when Mode 3 metrics are present.emit_backend_storage_routing_with_prometheusemitsengine: prometheus_remotefor Mode 3 metrics.deploy/configs/prometheus-otlp-receiver.ymlanddeploy/docker-compose/mvp-prometheus-archive.ymlfor the deployside.
Break-even table (50 B per raw sample)
KLL has no delta variant (per
Implementation.tex— randomisedcompaction is not additively mergeable).
Test plan
cargo test --release -p controller— 533 passed (vs. 516 baseline =17 new tests pass), 10 pre-existing failures unchanged.
(
cargo test --release phase_eps1→ 6/6 ok;cargo test --release planner::wire_cost→ 11/11 ok).with
--web.enable-otlp-receiver(separate PR).Notes for Phase ε.2
prometheus_remote.${ASAP_PROMETHEUS_QUERY_URL:-http://prometheus-archive:9090}/api/v1/query.time(orstart/end/stepfor range queries) → Prometheus → re-emit JSONresult back to the backend's HTTP query handler.
BackendAggregation::aggregation_input == Rawrequires an OTLP-rawingest path on
OtlpReceiverthat builds the sketch from samples onarrival (Mode 2; out of scope for ε.1).
🤖 Generated with Claude Code