validate Option B as sole bootstrap path; backend endpoint + accuracy reducer + freshness fixes - #332
Merged
Merged
Conversation
…rt + archive-tier accuracy ground truth This is the Option-B-as-sole-path validation re-run. With PR #329 (controller bootstrap GET → typed-stage-split) merged, the driver's post-stack-up `POST /api/v1/plan` workaround (PR #328) is redundant for the bootstrap path. Default it OFF so we can verify the bootstrap GET actually carries the load; operators can keep the safety belt by setting `ENABLE_OPTION_A_DRIVER_POST=1`. Three coupled fixes also land here so the demo's verdict matrix returns to all-green: 1. Backend container was crashing immediately on startup because `deploy/docker-compose/base.yml` passed `--ingest-port=9090` to a binary (`precompute_engine`, the deployed Dockerfile.backend target) that has no such flag — clap exited with `unexpected argument` and every subsequent Connection refused / failed POST from the controller cascaded from there. Drop the invalid flag and the redundant `19090:9090` host publish (Prometheus remote-write was deleted in ASAPQuery-backend PR #100; OTLP-only ingest goes over 4317 intra-docker). 2. `accuracy_reduce.py` is ported off the deleted `/var/asap/cold/raw/` JSONL tee onto the archive-engine-via-header path: `--cell-dir` re-issues each replay PromQL with `X-ASAP-Engine: thanos_archive` (Step 2.3 / PR #97) for ground truth. The driver's accuracy phase is rewritten to drop the `docker cp cold-truth/` step that was failing silently and skipping the entire reduce. The legacy JSONL path is kept behind `--use-jsonl` / `--use-jsonl-truth`. 3. STATUS detection in `capture_emitted_configs()` recognises the bootstrap-time typed marker (`[USE_TYPED_STAGE_SPLIT] emitted bootstrap config from typed path`) so a successful Option-B bootstrap renders as `live-bootstrap` even when the Option-A POST is gated off and `pushing typed` never fires. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 8, 2026
zzylol
added a commit
that referenced
this pull request
May 8, 2026
…g, warm + probe routing remain (#333) Changes: - controller typed bootstrap: extend Edge stage config with freshness probes + workload-registry archive metrics so the typed path emits gorillas3 + warm-passthrough routing (groundwork for OpAMP-on-connect future port) - thanos-store-gateway: shrink --sync-block-duration to 30s (was upstream default 3min) so demo soak window sees fresh blocks - mvp_report.py: surface real ⑥ pass/fail verdict from criterion eval, not placeholder CAPTURED/UNKNOWN rollup Verdict shift from PR #332's run: - ④ archive ground truth: 500 quantile rows archive_status=ok (was 0, all archive_miss) - ④ warm side: warm engine returns status=error for ALL 500 quantile queries — separate warm-engine pattern-matcher gap - ⑥ still UNKNOWN: agents load static placeholder via volume mount, NOT typed bootstrap; freshness-probe routing absent at agent - ① ② ③ ⑤: unchanged from PR #332 Two follow-up bugs documented: 1. Backend SimpleEngine errors on quantile_over_time and sum-instant queries (out of scope) 2. Static placeholder agent config has no routing for http_freshness_probe_* (out of scope; OpAMP-on-connect typed-config push is the durable fix)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ENABLE_OPTION_A_DRIVER_POST, default off) so the e2e demo runs Option B (PR controller: port handle_bootstrap_agent_config to typed-stage-split path #329's typedhandle_bootstrap_agent_config) as the sole controller emit path.--ingest-port=9090to a binary (precompute_engine) that has no such flag, so clap exited immediately on every backend bring-up. Removed the invalid flag and the stale19090:9090host publish (Prometheus remote-write was deleted in ASAPQuery-backend PR benchmark: q3 ground truth #100). The503 hot-reload handle not attachedfailure is fixed by ASAPQuery-backend PR benchmark: q8 ground truth #105 (separate repo).accuracy_reduce.pynow defaults toX-ASAP-Engine: thanos_archiveand the driver's accuracy phase no longer depends on the deleted/var/asap/cold/raw/JSONL tee.--use-jsonl-truthretained for legacy callers.[USE_TYPED_STAGE_SPLIT] emitted bootstrap config from typed path) so the verdict matrix renderslive-bootstrapwhen Option B carries the load.Verdict matrix (post-fix, both modes, soak=180s)
archive_miss— Thanos tier hadn't ingested data within the soak windowhttp_freshness_probe_*metric isn't surfaced through the warm/archive engine responselive-bootstrapnot-exercised§8 confirms Option B carries the typed bootstrap path: the controller's
[USE_TYPED_STAGE_SPLIT] emitted bootstrap config from typed path runtime=AsapOtel bytes=747log fires from the driver'scapture_emitted_configsGET against/api/v1/collector-config/agent, with Option-A POST disabled.Test plan
--no-cache;cargo check --bin precompute_enginecleanbash deploy/scripts/run_mvp_demo.sh --mode both(wall time ~25 min); driver waited on every background subprocess viawait $PIDrather thannohuplive-bootstrapstreaming-configPOST surface is wired (PR benchmark: q8 ground truth #105) — though Option B's bootstrap GET path doesn't itself push streaming-config (that's the Option A POST path the gate disables)🤖 Generated with Claude Code