Skip to content

mvp_report: ④ accuracy breaks out per sketch family (5 sub-rows + raw) - #337

Merged
zzylol merged 1 commit into
mainfrom
mvp/report-per-sketch-accuracy
May 8, 2026
Merged

zzylol merged 1 commit into
mainfrom
mvp/report-per-sketch-accuracy

Conversation

@zzylol

@zzylol zzylol commented May 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Refs #46. Renders ④ accuracy as 6 rows (5 sketch families + raw passthrough) instead of one aggregate, with per-family ε bounds and a single PASS/FAIL/UNKNOWN rollup in §2.

Per-family bounds (hardcoded; matches the shared MVP-demo contract):

Family Metric Query class Bound
DDSketch http_latency_ms quantile rel_err ≤ 0.01
KLL request_size_bytes quantile (rank-err) rel_err ≤ 0.005
HLL unique_users_per_min cardinality rel_err ≤ 0.0325
CountSketch top_endpoint_qps top-K recall ≥ 0.85
CountMinSketch endpoint_request_freq frequency rel_err ≤ 0.02 (≈ 1/w, w=64)
raw http_requests_total sum_rate exact (rel_err == 0)

Rendered ④ table shape

## §3 Accuracy (per sketch family)

| Sketch family | Metric | Query class | n | rel-err / recall | within ε bound? | Verdict |
|---|---|---|---|---|---|---|
| DDSketch | http_latency_ms | quantile | 300 | 0.0042 | yes (≤0.01) | PASS |
| KLL | request_size_bytes | quantile | 300 | 0.0030 (rank-err proxy: reducer reports value rel-err) | yes (≤0.005) | PASS |
| HLL | unique_users_per_min | cardinality | 300 | 0.0180 | yes (≤0.0325) | PASS |
| CountSketch | top_endpoint_qps | top-K | 300 | recall=0.920 | yes (≥0.85) | PASS |
| CountMinSketch | endpoint_request_freq | frequency | 300 | 0.0140 (additive proxy vs 1/w ≈ 0.02 ceiling) | yes (≤theoretical (1/w)) | PASS |
| raw | http_requests_total | sum_rate | 300 | 0.0000 | yes (exact (=0)) | PASS |

**Verdict ④:** PASS  · 6/6 families within bound; FAIL=0 UNKNOWN=0

Aggregate-verdict logic

The §2 row for criterion 4 reads Accuracy (per sketch family, see §3) and shows the rolled-up verdict:

  • PASS iff every family's observed value is within its bound AND raw is exact (== 0)
  • UNKNOWN if any family has no rows (bubbles up)
  • FAIL if any family is over-bound (bubbles up — UNKNOWN dominates over FAIL)

Detail summary in the §2 detail column: 6/6 families within bound; FAIL=0 UNKNOWN=0.

Honest caveats

  • KLL's reducer (accuracy_reduce.py) emits rel_err as value-relative-error, not rank-error. We compare it against the same 0.005 threshold and annotate the row as a "rank-err proxy". Fixing this requires the reducer learn to emit rank-err — out of scope for this PR.
  • CountMinSketch's additive-error bound (1/w against sum-of-frequencies) doesn't have a dedicated reducer column either; we reuse rel_err against a hardcoded 0.02 ceiling (≈ 1/w with w=64). Same fix path: extend the reducer.

Test plan

  • All 14 pre-existing test_mvp_report.py tests pass unchanged (one assertion updated to match the renamed §2 row).
  • 8 new tests added covering: all-pass, single-family fail, missing-family UNKNOWN, raw-must-be-exact, empty-csv UNKNOWN, table shape, single-mode end-to-end, dual-mode end-to-end.
  • Hand-rendered against a synthetic fixture covering all 5 sketch families + raw — output matches the contract.
  • Once a real run produces an accuracy.csv with all 6 families, verify the §3 table renders without manual fixture intervention.

🤖 Generated with Claude Code

Renders ④ accuracy as 6 rows (one per sketch family + raw passthrough)
instead of one aggregate. Each family carries its own ε bound:

  DDSketch       — rel_err ≤ 0.01
  KLL            — rel_err ≤ 0.005 (rank-err proxy)
  HLL            — rel_err ≤ 0.0325 (p=12)
  CountSketch    — recall  ≥ 0.85
  CountMinSketch — rel_err ≤ 0.02 (additive proxy, 1/w ceiling)
  raw            — exactly equal (rel_err == 0)

The §2 per-criterion table now shows ④ as a single PASS/FAIL/UNKNOWN
rollup (6/6 families within bound, etc.) with the per-family detail
rendered in §3. Aggregate verdict bubbles up: any UNKNOWN → UNKNOWN,
any FAIL → FAIL, otherwise PASS.

Caveats:
  - Reducer surfaces rel_err for KLL queries, not rank-err; we
    annotate the cell as a "rank-err proxy" until accuracy_reduce.py
    learns rank-err output.
  - CountMinSketch additive bound uses rel_err as a proxy vs a
    hardcoded 0.02 ceiling (≈ 1/w with w=64); reducer does not yet
    surface w or sum-of-frequencies-in-window.

Per-sketch metric → family contract is hardcoded in mvp_report.py
matching the shared spec used by the parallel agents touching
fake-exporter, mvp-workload.yaml, and the controller plumbing.

Tests: +8 new tests covering all-pass, single-family-fail, missing-
family-unknown, raw-must-be-exact, table-shape, and end-to-end
single-mode + dual-mode rendering against a fixture with all 6
families.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol merged commit d119add into main May 8, 2026
zzylol added a commit that referenced this pull request May 8, 2026
…equests_total_latency_ms) (#346)

PR #337's SKETCH_FAMILIES table used `http_latency_ms` as the DDSketch
metric, but the actual workload-spec entry (and fake-exporter emit) is
`http_requests_total_latency_ms`. Result: post-PR-#345 demo runs reported
DDSketch row n=0 even though warm tier was answering quantile queries —
the rel-err values were attributed to no family.

This aligns the table with the live workload-spec.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@zzylol
zzylol deleted the mvp/report-per-sketch-accuracy branch May 9, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant