mvp_report: ④ accuracy breaks out per sketch family (5 sub-rows + raw) - #337
Merged
Merged
Conversation
Renders ④ accuracy as 6 rows (one per sketch family + raw passthrough)
instead of one aggregate. Each family carries its own ε bound:
DDSketch — rel_err ≤ 0.01
KLL — rel_err ≤ 0.005 (rank-err proxy)
HLL — rel_err ≤ 0.0325 (p=12)
CountSketch — recall ≥ 0.85
CountMinSketch — rel_err ≤ 0.02 (additive proxy, 1/w ceiling)
raw — exactly equal (rel_err == 0)
The §2 per-criterion table now shows ④ as a single PASS/FAIL/UNKNOWN
rollup (6/6 families within bound, etc.) with the per-family detail
rendered in §3. Aggregate verdict bubbles up: any UNKNOWN → UNKNOWN,
any FAIL → FAIL, otherwise PASS.
Caveats:
- Reducer surfaces rel_err for KLL queries, not rank-err; we
annotate the cell as a "rank-err proxy" until accuracy_reduce.py
learns rank-err output.
- CountMinSketch additive bound uses rel_err as a proxy vs a
hardcoded 0.02 ceiling (≈ 1/w with w=64); reducer does not yet
surface w or sum-of-frequencies-in-window.
Per-sketch metric → family contract is hardcoded in mvp_report.py
matching the shared spec used by the parallel agents touching
fake-exporter, mvp-workload.yaml, and the controller plumbing.
Tests: +8 new tests covering all-pass, single-family-fail, missing-
family-unknown, raw-must-be-exact, table-shape, and end-to-end
single-mode + dual-mode rendering against a fixture with all 6
families.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 8, 2026
…equests_total_latency_ms) (#346) PR #337's SKETCH_FAMILIES table used `http_latency_ms` as the DDSketch metric, but the actual workload-spec entry (and fake-exporter emit) is `http_requests_total_latency_ms`. Result: post-PR-#345 demo runs reported DDSketch row n=0 even though warm tier was answering quantile queries — the rel-err values were attributed to no family. This aligns the table with the live workload-spec. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refs #46. Renders ④ accuracy as 6 rows (5 sketch families + raw passthrough) instead of one aggregate, with per-family ε bounds and a single PASS/FAIL/UNKNOWN rollup in §2.
Per-family bounds (hardcoded; matches the shared MVP-demo contract):
Rendered ④ table shape
Aggregate-verdict logic
The §2 row for criterion 4 reads
Accuracy (per sketch family, see §3)and shows the rolled-up verdict:Detail summary in the §2 detail column:
6/6 families within bound; FAIL=0 UNKNOWN=0.Honest caveats
accuracy_reduce.py) emitsrel_erras value-relative-error, not rank-error. We compare it against the same 0.005 threshold and annotate the row as a "rank-err proxy". Fixing this requires the reducer learn to emit rank-err — out of scope for this PR.rel_erragainst a hardcoded 0.02 ceiling (≈ 1/w with w=64). Same fix path: extend the reducer.Test plan
test_mvp_report.pytests pass unchanged (one assertion updated to match the renamed §2 row).🤖 Generated with Claude Code