Skip to content

benchmark: q8 ground truth - #105

Closed
GnaneshGnani wants to merge 11 commits into
debs_q7from
debs_q8
Closed

GnaneshGnani wants to merge 11 commits into
debs_q7from
debs_q8

Conversation

@GnaneshGnani

@GnaneshGnani GnaneshGnani commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Q8 — Price anomaly IQR fence (DDSketch precision/recall/F1)

Replaces the frac_iqr_lt_10pct metric with a Tukey fence anomaly detection evaluation: precision, recall, and F1 score for flagging stocks with price outliers. Sets accuracy_sla=0.0001 for Q8 (tighter DDSketch bucket width).

Benchmark results

Setup: Day 08-11-21, data_filtered, accuracy_sla=0.001, 4 × 15-min windows
Sketch: DDSketch [0.25, 0.75]; Tukey fence k=1.5
Thresholds: precision ≥ 0.70, recall ≥ 0.80, F1 ≥ 0.75

Window (CET) Valid IQR symbols Precision Recall F1
09:00–09:15 37.7% 0.776 0.450 0.570
09:15–09:30 37.3% 0.731 0.362 0.484
09:30–09:45 34.2% 0.653 0.278 0.390
09:45–10:00 30.3% 0.547 0.257 0.349
Aggregate 0.719 0.368 0.487

Fails — all windows fail recall threshold. Root cause: at accuracy_sla=0.001, ≈60–70% of symbols collapse to sketch_iqr=0 and are excluded; remaining symbols have overestimated IQR (2–3×), making the sketch fence too wide → borderline anomalies missed → low recall.

Next step: Re-run with accuracy_sla=0.0001 (already applied in run.py).

Cumulative: Q1: 11/12 ⚠️ | Q3: not run | Q4: 12/12 ✅ | Q5: 3/12 ⚠️ | Q6: 12/12 ✅ | Q7: 12/12 ✅ | Q8: 0/4 ❌

See results/benchmarking.md for multiplier sweep and full analysis.

@zzylol

zzylol commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Closing — this branch is conflicting with main (badly diverged, ~32k-line deletions) and predates the consolidated multi-sketch + latency eval harness merged in #494, which redid this ground-truth work. Please reopen and rebase if any ground truth here is still needed.

@zzylol zzylol closed this Jun 14, 2026
@zzylol
zzylol deleted the debs_q8 branch July 17, 2026 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants