Repository navigation
Add H200 profiling tables - #46
Merged
Merged
Conversation
Linear-op, attention and kernel-gap tables measured on NVIDIA H200 at TP1, eager and kernel-only (CUDA-event) families, in the per-model layout of the other devices. These are the tables the E2E calibration against vLLM 0.10.2 priced its dense co-location and PDD cases with.
Linear-op, attention, MoE and kernel-gap tables measured on NVIDIA H200 at TP1, with MoE rows at EP1 and EP2, eager and kernel-only families. These are the tables the E2E calibration priced its tiny-MoE PDD and DP/PP co-location cases with.
Attention rows measured on NVIDIA H200 at TP8 and TP2, the attention TP sizes of the two Qwen3-235B-A22B serving layouts of the E2E calibration. The table is committed alone so that each commit stays below 100 MB.
Kernel-only attention, linear-op, MoE and kernel-gap tables measured on NVIDIA H200 for the TP8/EP1 and TP2/DP2/EP4/PP2 serving layouts, joined into one per-model set: rows are told apart by their TP and EP columns. Both layouts' runs measured the TP1 linear-op rows; the set keeps the TP8 run's, following the merge rule that an earlier measurement of a row stays.
Host-time tables (h200_dgx) built from vLLM 0.10.2 CPU-probe runs. The predictor selects their rows by model and TP size only, so each model holds one serving layout: llama2_7b_dense_example co-location at TP1, Qwen3-30B-A3B-tiny PDD at TP1, and Qwen3-235B-A22B co-location at TP8 and at TP2/DP2/PP2.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
H200 profiling tables in the per-model layout of
data/profiling/compute/h800/andrtx_pro_6000/: one directory per model, with TP and EP as row attributes. Paths follow the default templates, so a run with--replica_config_device h200 --replica_config_network_device h200_dgxneeds no table flags.data/profiling/compute/h200/llama2_7b_dense_example/:linear_op,attention, their kernel-only tables,kernel_gap.data/profiling/compute/h200/Qwen3-30B-A3B-tiny/: the same plusmoeandmoe_kernel_only.data/profiling/compute/h200/Qwen3-235B-A22B/: the TP8 / EP1 and TP2 / EP4 table sets joined. Attention, MoE and CPU rows are disjoint by TP / EP; the 387 TP=1 linear-op rows measured in both sets keep the TP8 set's measurement.kernel_gap.csvholds a TP8 and a TP2 row per execution mode (needs the TP column of the base PR).data/profiling/cpu_overhead/h200_dgx/<MODEL>/: one serving layout per model. llama2_7b_dense_example: co-location TP1. Qwen3-30B-A3B-tiny: PD-disaggregation TP1. Qwen3-235B-A22B: co-location TP8 and TP2.Each commit stays under GitHub's 100 MB push limit (57.1, 69.1, 65.6, 58.0 and 0.2 MB of new blobs); the largest file is 65.6 MB.
Checks
data/profiling/, applies the kernel gap (eager 1.746 us, cuda_graph 0.394 us) and trains its CPU-overhead models. 4 requests of 128 prefill / 8 decode tokens read TTFT 27.059 ms, TPOT 26.697 ms (a path check; no vLLM reference).