phase 3.2.5: freshness probe routing — gorillas3 emit + warm passthrough + raw endpoint - #328
Merged
Merged
Conversation
…lanned Phase 3.2 (PR #320) diagnosed three freshness probe routing bugs. This commit fixes (a): the typed-stage-split path's emit_edge_yaml never included a `gorillas3` processor, so freshness probes (and any other archive-bound metric) never reached MinIO and the warm-tier engine's `last_over_time(...)` queries returned empty. Changes: * New `EdgeStageConfig::archive_tier_metrics: Vec<ArchiveTierMetric>` field. Populated by `ThreeStageEmitter` whenever a `RawAtEdgePrometheusArchive` node is encountered (per the spec: any Mode-3 metric also lands in the Gorilla-S3 archive so the warm-tier engine can serve `last_over_time(...)`). Callers that don't go through stage-split (the freshness probe path is the canonical example) populate the field directly. * New `ArchiveTierMetric` struct: `metric: String`, `window_secs: Option<u64>`. The L5 emitter uses the smallest declared window to size the `gorillas3.window_interval` knob. * `emit_edge_yaml` now emits a `gorillas3` processor block when `archive_tier_metrics` is non-empty. Config matches `deploy/configs/sketchcol-agent-b6-asap-single-sketch.yaml`'s canonical shape: `block_format: prometheus_tsdb` (so the Thanos store-gateway sidecar can read the blocks), `tsdb_bucket: asap-gorilla-tsdb`, `drop_original: false` (so the metric also flows downstream to the warm-tier sketch / OTLP exporter), all knobs env-overridable (`ASAP_MINIO_ENDPOINT`, `ASAP_GORILLA_BUCKET`, `ASAP_GORILLA_TSDB_BUCKET`, `ASAP_TENANT`). * The warm-tier pipeline lists `gorillas3` BEFORE the family-specific sketch processor — mirrors `sketchcol-agent-b6-asap-single-sketch.yaml`'s `[gorillas3, ddsketch, batch]` ordering so the cold-tier write happens on the raw sample BEFORE the sketch processor mutates / suffix-renames the metric stream. Mechanical updates: `stage_config_otap.rs` / `stage_config_telegraf.rs` / `config/mod.rs` add the new field to their literal `EdgeStageConfig` constructions (`Vec::new()`); these runtimes don't emit `gorillas3` themselves (Telegraf has its own `outputs.gorilla_s3` plugin path; the OTAP DAG runtime would land in Phase ε.1.5+1 if needed) so they leave the field empty. Tests: * `phase_3_2_5_bug_a_archive_tier_metrics_emit_gorillas3_processor` — locks in the gorillas3 block + every config knob the freshness probes need. * `phase_3_2_5_bug_a_gorillas3_runs_before_sketch_in_pipeline` — pins the processor ordering against `[ddsketch, gorillas3]` regression (which would lose freshness on the second-tier write). * Updated `phase_eps1_no_mode3_edge_yaml_unchanged_from_phase_b` to also assert `gorillas3` is NOT emitted when no archive tier is declared (preserves Phase β shape for plans without archive routing). Pre-existing 10 controller test failures unchanged. 537 tests pass (was 533 before; +4 new + 2 new postfix-runtime tests dragged in by the rebase). `cargo build --release -p controller` clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 3.2 (PR #320) diagnosed three freshness probe routing bugs. This commit fixes (b): the warm-tier DDSketch processor renames `http_freshness_probe_warm` to `http_freshness_probe_warm_quantile` (the family-specific `_quantile` suffix), so the replay client's `last_over_time(http_freshness_probe_warm[10s])` query never matches anything in the backend storage and ⑥ stays at FAIL. Picked the (b.1) routing path from the spec ("route around the sketch processor"), NOT (b.2) ("rename the probe to *_quantile"). (b.2) is the ugly-hack option — the freshness probe is a counter, suffixing it `_quantile` is misleading; (b.1) preserves the metric's design intent (the wire value `unix_ts_ms_of_emission` IS the freshness signal, exact-by-construction, not a sketch readout). Changes: * New `EdgeStageConfig::warm_passthrough_metrics: Vec<String>` field. The L5 emitter's caller populates this list with metrics that must bypass the family-specific sketch processor at the warm tier — the freshness probes are the canonical example. Other callers (every use site of the typed-stage-split path so far) leave it empty so the existing wire shape stays stable. * `emit_edge_yaml` now emits a `routing` processor with OTTL `route()` statements when `warm_passthrough_metrics` is non-empty. The processor table dispatches by `metric.name` (no need to teach the fake-exporter to set an extra attribute on top of the name) and folds the Phase ε.1 Mode-3 dispatch (`asap.mode == prometheus_archive`) into the same table when both axes are active. Three pipelines fan out: - `metrics/warm_tier` — gorillas3 + sketch processors → otlp/backend - `metrics/warm_passthrough` — gorillas3 → otlp/backend (NO sketch) - `metrics/prometheus_archive` (only when prometheus_archive_metrics is also non-empty) When `warm_passthrough_metrics` is empty the legacy `from_attribute: asap.mode` form is preserved verbatim so the existing Phase ε.1 wire shape is byte-identical (the `phase_eps1_mode3_edge_yaml_has_otlphttp_prometheus_exporter` test still passes against the legacy form). * Critical: the warm-passthrough pipeline still routes through `gorillas3` (when archive_tier_metrics is also configured) so the freshness probe lands in MinIO and the backend's GorillaQueryEngine can answer `last_over_time(...)` against it. Bypassing the sketch processor != bypassing the archive write. Mechanical updates: `stage_config_otap.rs` / `stage_config_telegraf.rs` / `config/mod.rs` add the new field to their literal `EdgeStageConfig` constructions (`Vec::new()`); these runtimes don't implement the warm-passthrough split (the OTAP DAG runtime would need its own routing-node analogue, ditto for Telegraf's processor pipeline) so they leave the field empty for now. Phase ε.1.5+1 follow-up if/when those runtimes need the same freshness-probe support. Tests: * `phase_3_2_5_bug_b_warm_passthrough_routes_around_sketch` — locks in the OTTL `route() where metric.name == ...` form, the `metrics/warm_passthrough` pipeline existence, AND the critical invariants: passthrough pipeline does NOT include `ddsketchprocessor` (the bug) but DOES include `gorillas3` (the archive write). * `phase_3_2_5_bug_b_warm_passthrough_composes_with_prometheus_archive` — pins the dual-axis dispatch (warm_passthrough + prometheus_archive together) so a single routing processor handles both. Pre-existing 10 controller test failures unchanged. 553 tests pass (was 551 after Bug (a); +2 new Bug (b) tests). `cargo build --release -p controller` clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 3.2 (PR #320) diagnosed three freshness probe routing bugs. This commit fixes (c): `run_mvp_demo.sh`'s `freshness_phase` was passing `PIPELINE_QUERY_PORT` to `run_freshness_phase.sh --raw-endpoint`, which resolves to the *backend* query port (`HOST_BACKEND_QUERY_PORT = 19091`) when running in asap mode. Per `mvp-freshness-probes.yaml` the raw probe target is `prometheus_b0` — the path's storage IS Prometheus B0, regardless of which pipeline is currently running: - http_freshness_probe_raw → routes to Prometheus B0 - http_freshness_probe_warm → routes to sketch warm tier - http_freshness_probe_archive → routes to GorillaQueryEngine The Step 2.4 endpoint table is unambiguous: baseline → 19090 (Prometheus B0) asap → 19091 (backend) For freshness/raw.csv, the polled endpoint is the BASELINE Prometheus B0 (port 19090). In baseline mode B0 is up under `--profile b0` and the probe lands there directly; in asap mode B0 is not running and the raw poll returns empty — that's the *correct* behaviour for the baseline-vs-asap comparison (asap doesn't carry a separate raw storage tier; the comparison's whole point is to show that). Change: --raw-endpoint "${ASAP_FRESHNESS_RAW_ENDPOINT:-http://localhost:${PIPELINE_QUERY_PORT}}" + --raw-endpoint "${ASAP_FRESHNESS_RAW_ENDPOINT:-http://localhost:${HOST_PROM_B0_PORT}}" Warm / archive endpoints stay on `PIPELINE_QUERY_PORT` — both paths route through the backend's storage-routing table to whichever tier the backend has wired (warm sketch in asap, b0 Prometheus in baseline). Verification: * `shellcheck deploy/scripts/run_mvp_demo.sh` — clean. * `shellcheck deploy/scripts/run_freshness_phase.sh` — clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 7, 2026
zzylol
added a commit
that referenced
this pull request
May 8, 2026
…rt + archive-tier accuracy ground truth (#332) This is the Option-B-as-sole-path validation re-run. With PR #329 (controller bootstrap GET → typed-stage-split) merged, the driver's post-stack-up `POST /api/v1/plan` workaround (PR #328) is redundant for the bootstrap path. Default it OFF so we can verify the bootstrap GET actually carries the load; operators can keep the safety belt by setting `ENABLE_OPTION_A_DRIVER_POST=1`. Three coupled fixes also land here so the demo's verdict matrix returns to all-green: 1. Backend container was crashing immediately on startup because `deploy/docker-compose/base.yml` passed `--ingest-port=9090` to a binary (`precompute_engine`, the deployed Dockerfile.backend target) that has no such flag — clap exited with `unexpected argument` and every subsequent Connection refused / failed POST from the controller cascaded from there. Drop the invalid flag and the redundant `19090:9090` host publish (Prometheus remote-write was deleted in ASAPQuery-backend PR #100; OTLP-only ingest goes over 4317 intra-docker). 2. `accuracy_reduce.py` is ported off the deleted `/var/asap/cold/raw/` JSONL tee onto the archive-engine-via-header path: `--cell-dir` re-issues each replay PromQL with `X-ASAP-Engine: thanos_archive` (Step 2.3 / PR #97) for ground truth. The driver's accuracy phase is rewritten to drop the `docker cp cold-truth/` step that was failing silently and skipping the entire reduce. The legacy JSONL path is kept behind `--use-jsonl` / `--use-jsonl-truth`. 3. STATUS detection in `capture_emitted_configs()` recognises the bootstrap-time typed marker (`[USE_TYPED_STAGE_SPLIT] emitted bootstrap config from typed path`) so a successful Option-B bootstrap renders as `live-bootstrap` even when the Option-A POST is gated off and `pushing typed` never fires. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 3.2 (PR #320) diagnosed three independent freshness probe routing bugs that prevent criterion ⑥ from flipping to PASS. This PR fixes all three in three isolated commits — the encoder is fine; the data-flow plumbing has the issues.
gorillas3processor → freshness counters never reach MinIO. NewEdgeStageConfig::archive_tier_metricsfield;emit_edge_yamlemits the canonicalgorillas3block (block_format=prometheus_tsdb, drop_original=false, env-overridable knobs) when set, ahead of the sketch processor in the warm pipeline._quantilesuffix renameshttp_freshness_probe_warm→*_quantileso the replay client'slast_over_time(http_freshness_probe_warm[10s])query never matches. Picked the (b.1) routing path:routingprocessor with OTTLroute() where metric.name == ...dispatches the probe to ametrics/warm_passthroughpipeline (gorillas3 still runs for archive write — ONLY the sketch processor is bypassed). Composes cleanly with Phase ε.1'sprometheus_archiveaxis when both are active.run_mvp_demo.shwas passingPIPELINE_QUERY_PORT(= 19091 in asap mode) to--raw-endpoint, contradictingmvp-freshness-probes.yaml's "raw → Prometheus B0" routing. Defaulted to${HOST_PROM_B0_PORT:-19090}.Test plan
cargo test --release -p controller— 553 pass / 10 pre-existing failures unchangedphase_3_2_5_bug_a_archive_tier_metrics_emit_gorillas3_processorphase_3_2_5_bug_a_gorillas3_runs_before_sketch_in_pipelinephase_3_2_5_bug_b_warm_passthrough_routes_around_sketchphase_3_2_5_bug_b_warm_passthrough_composes_with_prometheus_archiveshellcheck deploy/scripts/run_mvp_demo.sh— cleanshellcheck deploy/scripts/run_freshness_phase.sh— cleanfreshness/{warm,archive}.csvrows for the asap pipeline (raw stays empty in asap mode by design)Notes for Phase 3.3 (re-run demo)
gorillas3block honours:ASAP_MINIO_ENDPOINT,ASAP_GORILLA_BUCKET,ASAP_GORILLA_TSDB_BUCKET(defaults toasap-gorilla-tsdb, matches the Thanos store-gateway sidecar),ASAP_TENANT,ASAP_MINIO_ACCESS_KEY,ASAP_MINIO_SECRET_KEY. The compose overlay's existing MinIO credentials map directly.EdgeStageConfig::archive_tier_metricsANDwarm_passthrough_metricsfor the freshness probes — Phase 3.3's first task is to thread that population point (the bootstrap config emitter or the controller's per-stage push). Today'shandle_bootstrap_agent_config(legacy generate_agent_config) doesn't go through the typed-stage-split path — Phase 3.3 needs to either (i) port the bootstrap toemit_edge_yaml, or (ii) populate the two lists from the freshness-probe YAML at the controller-startup workload-registry pre-pop.mvp_report.pymay need a small tweak to treat asap-mode raw=∅ as "n/a" rather than "FAIL".🤖 Generated with Claude Code