benchmark: q3 ground truth - #100
Closed
GnaneshGnani wants to merge 42 commits into
Closed
GnaneshGnani wants to merge 42 commits into
GnaneshGnani wants to merge 42 commits into
Conversation
zzylol
added a commit
that referenced
this pull request
May 8, 2026
…rt + archive-tier accuracy ground truth (#332) This is the Option-B-as-sole-path validation re-run. With PR #329 (controller bootstrap GET → typed-stage-split) merged, the driver's post-stack-up `POST /api/v1/plan` workaround (PR #328) is redundant for the bootstrap path. Default it OFF so we can verify the bootstrap GET actually carries the load; operators can keep the safety belt by setting `ENABLE_OPTION_A_DRIVER_POST=1`. Three coupled fixes also land here so the demo's verdict matrix returns to all-green: 1. Backend container was crashing immediately on startup because `deploy/docker-compose/base.yml` passed `--ingest-port=9090` to a binary (`precompute_engine`, the deployed Dockerfile.backend target) that has no such flag — clap exited with `unexpected argument` and every subsequent Connection refused / failed POST from the controller cascaded from there. Drop the invalid flag and the redundant `19090:9090` host publish (Prometheus remote-write was deleted in ASAPQuery-backend PR #100; OTLP-only ingest goes over 4317 intra-docker). 2. `accuracy_reduce.py` is ported off the deleted `/var/asap/cold/raw/` JSONL tee onto the archive-engine-via-header path: `--cell-dir` re-issues each replay PromQL with `X-ASAP-Engine: thanos_archive` (Step 2.3 / PR #97) for ground truth. The driver's accuracy phase is rewritten to drop the `docker cp cold-truth/` step that was failing silently and skipping the entire reduce. The legacy JSONL path is kept behind `--use-jsonl` / `--use-jsonl-truth`. 3. STATUS detection in `capture_emitted_configs()` recognises the bootstrap-time typed marker (`[USE_TYPED_STAGE_SPLIT] emitted bootstrap config from typed path`) so a successful Option-B bootstrap renders as `live-bootstrap` even when the Option-A POST is gated off and `pushing typed` never fires. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
Closing — this branch is conflicting with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Q3 — Top-K frequency accuracy (CountSketch)
Fixes and runs the Q3 benchmark end-to-end.
Changes
run.py: Fix Q3QueryCfg— setgroup_by=("symbol",)(was empty) anddataset="data_filtered"(aligns with ground truth).controller/src/config/agent.rs: Emitdrop_originalfield in the generated collector YAML. Previously the field was never serialised, causing the CountSketch processor to default todrop_original: false, mixing raw financial metrics and sketch flushes in the JSONL output.compare.py: Inrun_comparison, detect mixed-content JSONL (raw batches + sketch flushes) and filter to onlycountsketch_partitionrows before per-window comparison. This was needed for the run before the controller fix, and remains as a safety fallback.results/benchmarking.md: Add Q3 results (5/12 windows pass).Results — day 08-11-21
q3_scorePass rate: 5/12 windows (threshold
q3_score ≥ 1.0).Analysis
The CountSketch estimates are accurate: Spearman ρ = 1.000 for 10/12 windows; total trade counts match ground truth within 0.4% per window. The overlap decline in later windows is a dataset property — after market open the frequency distribution flattens (3,100+ symbols all trading at ~300 events/5-min), making top-10 identification inherently ambiguous near the threshold. Window 1 has a clear 5× gap between window 1 and 11 (1,610 vs ~300 trades); by window 6 the top-10 range is only 300–313, meaning any one-count estimation error swaps symbols across the threshold.