bench(bw): expose the two things the hybrid-vs-offload ratio cannot see - #39
bench(bw): expose the two things the hybrid-vs-offload ratio cannot see#39gdevenyi wants to merge 1 commit into
Conversation
The verdict is a bandwidth ratio: CPU MoE GB/s vs PCIe gather GB/s. That answers
"which path moves expert bytes faster", which is not the question a deployment has.
Measured end to end on one box, hybrid beat offload by 16% at 10% GPU expert
residency and *lost by 33%* at 25% -- same hardware, same model, same ratio. The
benchmark reports one number for both.
This does not try to fix the verdict. It measures the two quantities the ratio
hides, and prints them:
* `pcie_gather_by_misses` -- gather bandwidth against the number of experts
actually missing per step. The headline figure refills a whole layer, which is
the best case for PCIe. I expected small gathers to be latency-bound and
therefore much slower; they are not. On a 12.75 MB expert it is linear from one
miss (24.7 GB/s) to 128 (25.0). Worth recording precisely because it rules the
PCIe side out as the source of the crossover.
* `cpu_moe_step_cost` -- one CPU MoE decode step split into a fixed cost and a
per-expert cost, by timing executors at `top_k = 1` and the workload's `top_k`
over the same banks. ds_fp4 on this box: 0.08 ms fixed + 0.19 ms/expert. The
fixed part -- activations D2H, the GPU<->CPU handshake, waking the pool and
draining its barrier, results H2D -- is paid per layer per step whether one
expert misses or twenty, and the GPU stalls on it. Across 43 layers that is
~3.4 ms/step of toll that the bandwidth ratio does not model at all.
Deliberately NOT included: a derived break-even miss count. The obvious closed form
(fixed / bytes-saved-per-expert) says hybrid should win above ~0.3 misses, i.e.
essentially always -- which contradicts the measured end-to-end crossover. Something
else is going on (imperfect overlap, GPU stall on the handshake, non-uniform miss
distribution under LRU), and shipping a tidy formula that disagrees with the one
real measurement available would be worse than shipping nothing. These stay
diagnostics until an end-to-end harness can calibrate them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpGe2fQ5pDShGrnSuksfun
|
The paper (arXiv:2608.16157 §3.2) turns out to name exactly the gap this PR measures, which makes a better case for it than I made originally. Equation 3 gives the two branch times:
and Equation 4 balances them to q* ≈ m · B_P/B_H.
That is also why §3.2's model predicts hybrid should win essentially always: as B_H → B_P the policy degenerates gracefully to pure cache fill, so the natural crossover sits at a bandwidth ratio of 1.0, not the 2.0 threshold the implementation uses. The implementation's conservatism looks justified — I measured hybrid losing end to end at a ratio of 2.5× (offload 16.72 vs hybrid 16.07 tok/s single-stream, 67.11 vs 56.57 at 8 concurrent streams). For completeness, adding the measured constant to Eq. 3 still does not reproduce that crossover. Break-even is
does not hold in practice, and the GPU partly serialises behind the CPU branch on the handshake rather than overlapping with it. If exposed latency is closer to a sum than a max, the whole balance changes. That is a hypothesis, not a measurement, which is why this PR still ships the two quantities and no derived verdict. It does suggest where to look next: instrument the exposed-vs-sum question directly, since it would invalidate Eq. 4 rather than just correct it. |
|
Re-tested against current main ( Method. This PR's head merged onto main, then the full Result: 1205 passed, 350 skipped, no new failures. 🤖 Generated with Claude Code |
1 participant
The problem
The CPU-to-PCIe ratio answers one question: which backend is faster per byte. Two things it cannot see also decide what a deployment gets.
The first is the miss rate. The offload path only pays PCIe cost for experts that are not resident. A cache that holds most of the working set moves the crossover a long way, and the ratio does not know the cache size.
The second is the fixed cost of a CPU MoE step. Below some batch size, a step costs about the same regardless of how few bytes it moves, so the per-byte ratio overstates what the CPU path delivers.
What this adds
Two measurements, reported alongside the existing ratio:
measure_pcie_gather_by_missessweeps the miss count and reports gather cost against it, so the crossover can be read off directly rather than assumed.measure_cpu_moe_step_costreports the fixed per-step cost of the CPU path.Neither changes the verdict. They are there so the number that does drive the verdict can be checked against the two effects it ignores.
Testing
tests/moeonmainwith this PR: 94 passed, 6 skipped, 1 failed. The failure istest_cpu_moe_q4_0.py::test_cpu_decode_q4_0_matches_ggml_mmvq, which also fails onmainwithout this PR.This PR adds no test of its own. Both additions are timing code, so asserting values would be flaky, but their shape is testable: gather cost should rise with miss count, and step cost should flatten below a batch threshold. Tell me if you want that before merge.