bench(bw): repeat each measurement and withhold a verdict that straddles the threshold - #37
Open
gdevenyi wants to merge 1 commit into
Open
bench(bw): repeat each measurement and withhold a verdict that straddles the threshold#37gdevenyi wants to merge 1 commit into
gdevenyi wants to merge 1 commit into
Conversation
…les the threshold
`ft bench bw` measured each kernel once and printed a single ratio with a single
backend recommendation. For a stable format that is fine. For one whose kernel is
compute-bound, or on a box with any background load, it is not: mxfp4 was observed
between 19 and 71 GB/s on the same machine, flipping hybrid<->offload between
invocations while every individual run reported one confident number.
Repeat each kernel measurement (`--reps`, default 3), take the median, and bound
the ratio by the worst and best pairing actually observed. When that interval
clears the threshold the call stands; when it straddles it, say so and fall back
to `offload` -- the backend that always works, and the way `bench_profile` already
resolves an ambiguous profile.
Real output, same box, one invocation:
bf16 9.00 MB 69.6 GB/s 25.1 GB/s 2.77x hybrid
3 runs: CPU-MoE 68.8-69.7, ratio 2.74-2.77x
mxfp4 12.62 MB 44.4 GB/s 25.2 GB/s 1.76x offload
3 runs: CPU-MoE 42.2-50.2, ratio 1.68-2.00x
Same table, and now the reader can see that one of those numbers is worth
believing and the other is nearly a coin toss.
The profile gains `cpu_moe_gbs_runs`, `pcie_gather_gbs_runs`, `ratio_range`,
`reps` and `confident`. `recommended` keeps its existing domain
("hybrid"/"offload") so `bench_profile.load_backend_recommendation` and any older
reader are unaffected -- an unstable format simply reports the conservative value.
The decision is extracted into a pure `verdict()` so it can be tested without a
GPU; `tests/moe/test_benchbw_verdict.py` covers clear-hybrid, clear-offload, the
straddling case, the single-rep path (unchanged behaviour) and the exact-threshold
boundary.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpGe2fQ5pDShGrnSuksfun
This was referenced Aug 23, 2026
Author
|
Re-tested against current main ( Method. This PR's head merged onto main, then the full Result: 1210 passed, 350 skipped, no new failures. The +5 over main are this PR's own tests, and they ran (not skipped). 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
ft bench bwmeasures the CPU MoE kernel and the PCIe gather once each, divides them, and prints a backend verdict. The ratio decides whether a deployment should runhybridoroffload.One measurement of each is not enough to support that decision. On this machine the CPU MoE figure for bf16 moves between about 61 and 70 GB/s across repeats of the same command. With a threshold at 2.0x, a true ratio near 2.0 will print
hybridon one run andoffloadon the next, and neither is wrong.What this changes
--reps Nrepeats each measurement and reports the median with its range. When the range straddles the threshold, the tool withholds the verdict instead of printing one, and says which side the median fell on.The default is 3, so existing invocations get repeats without a flag change.
Why it matters beyond tidiness
I used this on my own PR #45 and it changed the outcome. One pair of runs showed a 6.9% regression from that PR, and a different pair showed a 1.9% gain. With repeats the two arms plainly overlap, so the honest answer is that the change does nothing measurable on this machine. Without repeats I would have reported whichever pair I ran first.
Testing
tests/moe/test_benchbw_verdict.pycovers the verdict as a pure function: clear hybrid, clear offload, and a straddling range that must withhold. Onmainwith this PR: 5 passed.tests/serveronmainwith this PR: 540 passed.