Repository navigation
calib: price host, DP/PP and EP execution against measured vLLM 0.10.2 (H200 gate 21/22) - #44
Merged
Merged
Conversation
The case_002 native forward takes about four times Frontier's price, and neither existing mode can say where the time goes: `clean` has no operator records and `instrumented` adds a host sync per layer for the MoE routing log. `op_timing` and `kernel_timing` turn on CUDA-event timing per operator scope, matching the CUDA_EVENT family of the Frontier profiles, together with the batch, PP boundary and schedule records. `op_timing` times each scope as it runs; `kernel_timing` syncs the device before each scope so that a scope holds kernel time only. The MoE routing log stays off in both. The placement extractor now separates the DP dummy-pass records, which the ground-truth checkout writes after every step that executes no model, from the engine iterations, and leaves the E2E request metrics columns empty for a mode that does not write them. Re-extracting the case_002 clean run gives the same placement, iteration and snapshot tables.
`op_gap_analysis.py` reads a case's clean, op (`default` scope mode) and kernel (`kernel_only`) native runs and writes the tables the I4 analysis needs, without drawing conclusions: - batch windows joining op_timing, batch_log and pp_boundary per (pp_rank, dp_rank, batch_id), with every missing, repeated or disagreeing key reported in the summary; - scope and window distributions per shape class, and the op-run minus kernel-run time per scope; - the peer dp_rank's state (batch, DP dummy pass, idle) over each batch; - DP dummy pass durations; - the decode-cycle split of the clean and op runs and, per (running, other_running) class, the op-run median less the clean-run median, the measurement-cost estimate. Cycles before the first formal admission are warmup and are counted, not used; - the op run's scheduling sequence aligned against the clean run's, and the per-request engine placement compared across runs. The engine to dp_rank mapping is derived from request ids and checked, and the tool tests that engine and pp_boundary timestamps share a clock.
…cases The stack accuracy gate compares four cases (dense and MoE, co-location and PDD) with clean vLLM runs. Until now each case carried its own copy of the E2E gap script. This tool generalizes the dp_pp_case_002 script: - several Frontier runs of one case in one table; the first is gated, the others (for example origin/main) are references that never change the verdict; - a request-id gap (missing, duplicated, or in no id list), a token-count mismatch, or a MoE routing status other than MATCH blocks the gate, as the calibration contract requires; - relative error is undefined when the vLLM value is zero; - a metrics file older than the file its run writes first is rejected, so a leftover from an earlier run cannot enter the table. On the case_002 inputs it reproduces both generation-2 tables (d7e41ee and 722ce68) cell for cell, and the normalized rows are identical.
The gate measures the stack head (fix/vllm-v1-skipped-requeue-order at 4a2a4e3). The case harness under tests/comparison/dp_placement_pp/ lives on the stack, so ticket 01 builds on it.
…_trace The E2E accuracy gate cases (C1 dense co-location, C3 dense PDD, C4 MoE PDD) replay issue #26's prefill-heavy workload: 100 requests of 2048/256 tokens with Poisson arrivals at QPS 2.0. Their warmups must cover the formal shapes: isolated 2048-token prefills, then a burst that reaches the formal decode batch. - `warmups` may list named `groups`, each with its own lengths and spacing and a `gap_before_s` after the previous group; a group's request ids carry its name. - An optional `poisson` segment sends `count` requests at `qps`: the first at once, each later one after an exponential interval from random.Random(seed), as a serving benchmark sends them. - `burst`/`bursts` and `steady` become optional. A workload without these keys builds the same trace as before: dp_pp_case_002's trace.csv and request_ids.json regenerate byte-identical (trace SHA-256 540e5bac...). test_warmup_groups_precede_seeded_poisson_arrivals covers the new keys.
… transfer The PD cases of the stack E2E accuracy gate (C3 dense, C4 MoE) need values for Frontier's analytical KV-cache transfer (bandwidth and latency) on one H800 node, and no existing measurement covers them. nccl_p2p_bench.py sends from GPU 0 to GPU 1 over an NCCL group and times two patterns with receiver CUDA events: a single-message size sweep, fitted to latency + bytes / bandwidth, and the per-request pattern of vLLM's P2P NCCL connector in PUT_ASYNC mode (one message per layer, sized by the simulator's own KV layout). It maps the result onto network_bandwidth_gbps and network_latency_ms; the latency is a lower bound, since the connector adds a ZMQ handshake per tensor. The unit test checks the message sizes, the fit, and that Frontier's AnalyticalKVCacheTransferPredictor reproduces the measured request time from the mapped values.
…ctor sends vLLM's P2pNcclEngine synchronizes its send stream after every per-layer message (send_sync) and creates its communicator with NCCL_MAX_NCHANNELS and NCCL_MIN_NCHANNELS set to nccl_num_channels and NCCL_CUMEM_ENABLE=1 (set_p2p_nccl_context). The benchmark sent the layer messages back to back under NCCL's own channel choice, so its request time and the Frontier KV transfer values derived from it would come out faster than the connector. The sender now synchronizes after each message, and both ranks set the connector's NCCL variables from the new required --nccl_num_channels before the process group creates its communicator. The summary records the variables. The ZMQ handshake per message is still not reproduced, so the request time stays a lower bound.
…gate cases The PD cases of the stack E2E accuracy gate (C3 dense, C4 MoE) compare Frontier's pd-disaggregation with a vLLM prefill instance and decode instance joined by P2pNcclConnector in PUT_ASYNC mode (decisions Q4, Q5, Q13). vLLM's example proxy cannot be joined to trace rows (random request ids), skips a failed prefill, and needs quart, which the image lacks. - pd_proxy.py: the example's flow (prefill copy with max_tokens=1, then the unchanged request to decode) on aiohttp, with the client X-Request-Id in the proxied id, min_tokens=1 on the prefill copy, a 502 on a failed prefill or connection, and one timing record per request. - pd_replay.py: starts both `vllm serve` instances with static connector addresses from the engine file's kv_transfer block, replays the trace through the proxy with vllm_replay.replay, and fails when a server log lacks the PUT_ASYNC init line or holds a connector fault line. - pd_metrics.py: joins proxy, prefill and decode records into one row per request under the Q13 metric contract, and lists every request it cannot join. - run_pd_worker.sh: the worker, with the environment of run_vllm_worker.sh. - e2e_metrics_gap.py reads those joined rows for a pd-disaggregation case.
…ory filter The attention wrapper gives every sequence of a true-mixed batch its own KV blocks, rounded up to whole blocks, but the input filter compared the batch's token total against max_num_blocks * block_size. A batch could pass the filter and then fail in the wrapper, which ended the whole attention run: the H800 dense run for the E2E gate stopped at three 2048-token prefills plus 64 decodes at KV 2176 (145,472 tokens, 9152 blocks against 9122). The filter now sums ceil(total_len / block_size) over the sequences, the same count the wrapper allocates. For the gate's dense grid it drops exactly that one input (1615 -> 1614 of 1680). The simulator does not import this module.
…s hit - vllm_replay.server_env sets VLLM_V1_ALLOW_NO_CHUNKED_PREFILL=1 when the engine file disables chunked prefill. vLLM-BS refuses V1 generation without chunked prefill unless it is set, so C1, C3 and C4 (chunked prefill off, decision Q20) exited at server start. The inherited value is dropped, so the engine file alone decides it. - run_profiling_plan runs an invocation by `script` path as well as by `module`. The NCCL benchmark lives under tests/, which has no __init__.py; the vLLM image installs its own `tests` package, which shadowed it for `python -m` (No module named 'tests.comparison').
…ll prefill once get_attention_input_combinations adds the first chunk (kv 0, batch 1) of every chunk size and a full prefill of every length, so a chunk size that is also a full-prefill length produced the same input twice. Without the chunk grid search that is the max_seq_len prefill; with it, every chunk size. The E2E gate MoE attention CSV from H800 had rows 2 and 82 with the same identity, and the profiling CSV merge rejects a repeated identity. The combinations are now deduplicated in first-seen order before validation. The simulator does not import this module.
…-e2e-gate Brings the operator-timing ground-truth modes (op_timing, kernel_timing) and the native op-gap tool into the gate branch, for the C1 TPOT/E2E RCA. The mode test now gives the timing modes an engine with enable_chunked_prefill, which server_env reads since 5660cb4.
…modes The fork's default scope list has no mlp_up_proj, mlp_act or mlp_down_proj, so an op_timing or kernel_timing run of the dense Llama case would leave about half of each decode step untimed. The timing modes now pass VLLM_FRONTIER_CUDA_EVENT_OP_SCOPES: the scopes the Llama and Qwen3-MoE models open in the calibration cases. Against the fork default this adds the three MLP scopes and drops the MLA and TP/EP all-reduce scopes, which no case reaches (TP=1, EP by all-to-all); the MoE case_002 scope set is unchanged.
The standard decode and multi-prefill samples gave every sequence the same block table, so a decode batch read one sequence's KV pages for the whole batch. The profiled decode medians implied read rates above the H800 HBM peak (3.9-7.7 TB/s at KV 2240, batch 8-64), and Frontier's decode attention grew about 4x slower per extra sequence than vLLM's (C1 error analysis, source F2; correction R1 approved in review). - All three sample builders now allocate consecutive, disjoint block ranges per sequence through one allocator, as the true-mixed builder already did and as the engine's paged cache does. - The memory filters count whole blocks per sequence against the block budget instead of tokens, matching the allocator. - In kernel-only (record_function) profiling, the wrapper reads a buffer of twice the device L2 size between the warmup and the traced call, so the traced call reads the KV cold, as each engine layer does.
Correction R2a runs the gate cases with decode CUDA graphs on the vLLM side (FULL_DECODE_ONLY), so the harness must pass the engine's graph setup and must record per-step timing without the operator probes. - server_command passes the engine file's compilation_config to --compilation-config as given; without it vLLM keeps its default. - New ground-truth mode schedule_timing: only the scheduler's host-side per-step log (VLLM_FRONTIER_SCHED_LOG_PATH) is on, so its step timestamps describe an otherwise clean engine. - The PD harness reuses server_command and server_env unchanged.
vLLM's decode CUDA graphs replay a FlashInfer decode plan made with use_cuda_graph=True for the padded graph batch (split-KV forced, grid sized for that batch), while Frontier's attention profiler uses the eager plan for the real batch. The bench times the decode kernels of both plans on the same per-sequence KV pages, cold in L2, with the profilers' kernel-only method, so a profiler change is proposed only on a measured difference (ticket 14).
The calibration contract's CPU mode enables the CPU probe alone, with operator instrumentation and E2E request metrics off. vLLM-BS 576826cc4 adds that probe (VLLM_FRONTIER_CPU_PROBE_LOG_PATH, host-only per-step timestamps written at engine shutdown). The replay harness now sets it in mode cpu, for both the co-location server and the PD instances, so the E2E gate can attribute the per-step host time that Frontier's decode steps are missing.
The independent training path (co-location without the shared model manager) trained the kernel-only family's CPU-overhead models from the eager cpu_overheads.csv, because activating a measurement family swapped the compute, attention and MoE inputs but not the CPU-overhead input. The shared manager already reads cpu_overheads_kernel_only.csv for graph-replayed decode steps through resolve_measurement_input_paths. Activating a family now also selects its CPU-overhead file: KERNEL_ONLY reads cpu_overhead_kernel_only_input_file (as resolved, with its existing fallback), CUDA_EVENT and DEVICE_EVENT read cpu_overhead_input_file. Runs with skip_cpu_overhead_modeling (the default) are unchanged. Ticket 19 of the PR-stack E2E accuracy gate (decision: co-location CPU family fix).
… batch A FULL_DECODE_ONLY graph replay pads the decode batch to the next capture size. Frontier looked decode attention up at the padded batch size, which prices every padded slot as a real sequence at the batch's average context. vLLM's padded rows hold no KV pages, so the attention kernel works on the real decode requests only. Against the forward device time of the C1 and C3 CPU-probe runs this overpriced padded steps by 0.28 to 2.17 ms per step, while captured sizes matched within 1.8%. Decode attention now uses the number of decoding requests for every step. The two get_effective_decode_batch_size_for_attention methods had this predictor path as their only consumer and are removed. Linear operators still use the padded token count, and padded_decode_batch_size stays for the CUDA-graph activation records. Ticket 18 of the PR-stack E2E accuracy gate (decision R3-P, Part 3).
Frontier's CPU-overhead tables priced vLLM host work from Sarathi-era profiles, so the E2E gate cases had no CPU-overhead data measured on the engine they compare against. The instrumented vLLM already writes one CPU-probe record per step with host timestamps and the forward's CUDA-event device time. vllm_cpu_probe maps each step to the four schema terms (schedule, prepare_inputs_e2e, sampler_e2e, process_model_outputs) so they add up to the step period minus the forward device time, groups steps by the (batch_size, num_prefill_tokens, num_decode_tokens) identity, and writes one validated CSV per measurement family: pure decode tuples within the decode CUDA-graph capture sizes go to the kernel-only file, the rest to the eager file. Verified on the C1 and C3 CPU runs: every tuple's medians equal the combined five-term analysis (max |diff| 4.4e-16 ms).
…e-request successor A PD prefill instance runs each request for one step, so no step there is followed by a step of the same requests and the producer wrote no eager rows for the C3 and C4 prefill tuples. When the next step shares no request, the engine may have waited for an arrival, so process_model_outputs now ends at update_end: the measured host work after sampling, leaving out only the engine loop gap (busy medians 0.02-0.05 ms on the gate cases). Steps with a same-request successor keep sample_end -> next step_start, and every tuple now carries all four terms, so the skip path is removed.
The shared trainer always ran GridSearchCV and asked for two folds when a table had one row, which scikit-learn rejects. C4's PDD PREFILL CPU-overhead table has a single measured tuple (1, 2048, 0), so the simulator aborted during predictor initialization. Every grid candidate fitted on one row predicts that row's value, so the estimator is now fitted directly; tables with two or more rows keep the grid search with cv = min(k_fold_cv_splits, rows).
kv_save_timing sets the CPU probe and the producer connector's KV-save probe of a vLLM-BS diagnostic commit; kv_save_device_ids also switches the connector to a device copy of the block ids. Both serve the C4 TTFT residual attribution (E2E gate ticket 22). The inherited-switch strip keeps them off in every other mode.
…atch C5 (Qwen3-235B-A22B, TP8) needs Frontier's analytical TP all-reduce parameters from the collective the engine actually runs (decision C5-AR, ticket 24). The existing collectives profiler calls torch.distributed directly and so bypasses vLLM's CudaCommunicator, which sends decode-size messages through custom all-reduce and prefill-size messages through NCCL. The bench joins vLLM's distributed state on the GPUs of one node and calls tensor_model_parallel_all_reduce on hidden-state messages of the model: decode sizes captured in one CUDA graph under graph_capture, as the FULL_DECODE_ONLY graphs are, and step sizes called eagerly. Each size records the dispatch path vLLM chose. The fit takes the bandwidth from the eager NCCL slope and the latency from the graph custom all-reduce rows, and records every size's relative error against the fitted time.
Ticket 25 (decision A+B) runs the ground truth at TP8 on vLLM-BS d524a142e, where only TP rank 0 writes the batch, operator-timing and PP boundary logs and each record carries tp_rank. vllm_replay.py now requests the four TP all-reduce scopes the fork opens at TP > 1 (attn_post_proj_tp_allreduce, mlp_down_proj_tp_allreduce, moe_tensor_parallel_allreduce, tensor_parallel_allreduce); at TP1 none opens, so the TP1 scope set is unchanged. op_gap_analysis.py counts them as collectives. The first two run inside attn_post_proj and mlp_down_proj, so their time is taken out of the non-collective sum instead of being counted twice. With one writer per batch, (pp_rank, dp_rank, batch_id) stays the batch key and tp_rank is carried as an extra column. The unit test rows carry tp_rank, and a new test checks the nested all-reduce is counted once.
…family An EP lane batch prices one MoE layer of its source forward step. Its logical requests are the lane's local experts, so it counts every routed token as prefill and carries no decode CUDA-graph metadata. In a co-location MoE decode step replayed as a CUDA graph, the selector therefore priced the MoE layers from the eager table while the rest of the step used the kernel-only table. Qwen3-235B-A22B (C5, TP8, EP1, FULL_DECODE_ONLY) is the gate case where this happens. The predictor now reads the decode state and the graph runtime mode from the lane's source step batch, both when it selects the measurement family and when it decides whether to strip the collective-sim all-reduce launch overhead. Specialized roles return their cluster default before reading the batch, so only MONOLITHIC selection changes; the strip check now also applies to DECODE MoE lanes of graph steps under collective_sim. CUDA-graph activation records still read the lane itself, which has no metadata, so no duplicate records are emitted. Ticket 28 (C5 correction R1, option B).
…E routing A ground-truth engine file with moe_uniform_routing set now starts the vLLM server with VLLM_MOE_UNIFORM_ROUTING=1, and an inherited value no longer leaks into the server. The C5 ground truth needs vLLM to route like Frontier's balanced distribution; the engine file is where each run's engine settings are pinned and recorded.
A decode CUDA-graph step runs the whole model at the padded capture size, FusedMoE included (vLLM pads num_input_tokens and passes the padded input to the model). Frontier already priced attention, linear ops and the pre-routing MoE operators at Batch.get_effective_total_tokens_for_compute, but the routed expert workload was materialized from total_num_tokens. Against the C5 round-robin ground truth this left decode steps short right above capture boundaries (bs9 -25.0%, bs10 -20.2%, bs5 -14.9%), while vLLM's period is flat within a capture size. Both routed-workload sites, the scheduler's materialize_layer_workload and the predictor's _materialize_layer_ep_workload, now take the same compute width. Steps without decode graph metadata (eager, prefill, mixed), DP>1 aggregates and PD-AF DECODE_FFN lanes keep their widths. Ticket 29 commit 1 (decisions G1, G2); change marker t29-routed-moe-width-follows-graph-padding-20261001.
An eager forward step lasts as long as the slower of two streams: the device running the forward's kernels and the host launching them. Frontier priced eager steps from eager single-operator timings, which include each kernel's launch latency. For small TP8 operators those timings are launch-bound, so a device-bound step such as C5's 2048-token prefill (vLLM: host launch 91.6 ms, device 119.3 ms) was overpriced. When the eager CPU-overhead table carries the new forward_launch columns (host time from preprocess_end to forward_end, published by the vLLM CPU probe), eager steps on co-location, PDD PREFILL and non-graph DECODE roles: - take operator times from the kernel-only tables (device stream); - keep their CPU-overhead terms from the eager table, forward_launch included; - add forward_launch_stall_time = max(0, forward_launch - device span) to the step wall, with each wall site using its own span: model_time on the BatchStage path, and (sync - stage start) + pipeline on the MoE collective path. Graph decode steps and tables without forward_launch keep their pricing. PD-AF and num_pipeline_stages > 1 fail fast on such a table. The probe now publishes sampler_e2e net of max(forward_launch, device time) for eager steps, so the step's window is not counted twice. The kernel-only family trains prefill and true-mixed attention models when its table has those rows. The shared manager trains and exposes the kernel-only family for two-stream clusters, and its per-cluster model view now derives from the same family list used for training.
vLLM runs the schedule call and the input preparation of a step before the forward's DP token-count all-reduce, so a lane prepares its inputs while the device still drains earlier work and while a later DP lane catches up. Frontier started the forward's first layer at the stage start and added the preparation after the last layer, in series with the DP layer syncs. The stage now launches its forward at stage start plus the schedule and input-preparation time (OverheadTime.forward_preparation_time), the first Prefill/DecodeSyncEvent fires from that launch start, and the final stage timing leaves the preparation out of the CPU tail. The total CPU overhead of a step is unchanged; DP = 1 runs differ only by float rounding. Per-mode readings against vLLM v0.10.2 on H200: C2' split TTFT error -13.2% -> -2.6% (qps2) and +9.2% -> -4.3% (qps8, TPOT +4.0% -> -1.3%); C6 qps3.5 split TTFT -10.5% -> -7.4%.
With engine-idle buckets, an engine's first step that schedules tokens has no earlier step or dummy forward to measure idle time from, so it went to the largest bucket. That step also carries the engine's one-time start costs, which a ground-truth run's warmups absorb: on the C6 H200 CPU probes, the 2048-token rows at 500 ms idle came from three such steps (stage-0 prepare 28.8 ms against 2.3 ms warm, launch 122.7 against 68.2 ms). Frontier then priced the first formal prefill after a long idle gap about 80 ms above vLLM, which shifted the DP lanes' phase and the cell's mode. The step's terms are still computed, so the next step's dispatch queueing keeps its forward end; it is only left out of the rows. Tables without the engine_idle_ms column keep every step, as before.
…ling step The engine-idle CPU-overhead rows stopped keying an engine first token-scheduling step as the largest idle edge, but the profiling guide still described the old rule. State the current behavior: that step carries the engine one-time start costs that a ground-truth run warmups absorb and is left out of the rows, its terms are still measured so the next step keeps its own dispatch queueing, and every step is kept when no idle edges are requested.
…to the calibration chain Brings the three tests/comparison/dp_placement_pp/ changes the ground-truth runs depend on into one branch, so the chain and the harness that measured it ship together: the replay event-loop wake (1a418c0), the shared Triton kernel cache per ground-truth job (03699e2) and inclusive warmup length ranges in make_trace.py (118cc79).
…treams uses_two_stream_eager_pricing switched eager steps to kernel-only operator tables whenever the eager CPU-overhead table carried forward_launch for every stage, without checking that the rows belonged to the replica's model and attention TP degree. A table with launch rows for TP2 only enabled two streams on a TP1 replica; CPU-overhead training then found no rows and the forward_launch lookup priced the host stream as 0. An estimator that failed validation was also turned into 0 by the lookup's catch-all (review C-44-1). The selector now takes the replica config and requires forward_launch rows for its model and TP degree at every pipeline stage, failing with the missing model/TP otherwise, before any operator family changes. forward_launch is the whole host stream of such a step, so its lookup is required: a missing model, a missing batch-size row or an invalid estimate raises. Optional terms keep pricing 0 with a warning.
The cross-DP aggregate of an EP wave summed the lanes' request tokens and then overwrote the sample lane's decode-graph padding with that sum. Two lanes of nine decode rows, each run at capture size 16, routed 18 tokens (36 assignments at top-k 2) instead of 32 (64), and a lane paired with a dummy pass routed 10 (review C-44-2). vLLM runs each DP rank's forward at its CUDA-graph capture size, or eager at its own tokens, and unless the engine is eager pads every rank, a dummy pass included, to the largest of those widths (gpu_model_runner get_dp_padding). prepare_ep_wave_inputs now takes each lane's compute width from get_effective_total_tokens_for_compute and, when the lanes carry decode CUDA-graph metadata (the role models graphs), pads every lane to the largest; eager roles keep the sum of their own tokens. The predictor-only aggregate carries that width into routing, EP lanes and prediction. The source lane batches keep their request tokens. Lane-local dense work is still priced at each lane's own width.
…38-1, C-40-1, C-42-1 and C-42-2 fixes
vLLM applies a batch's output (Scheduler.update_from_output) only when an engine iteration pops it from the batch queue, and that iteration schedules before it pops. When a batch ended while its host waited in the DP finish sync, Frontier kept the queue slot but had already committed the batch's request state, freed finished requests' KV blocks, recorded completion metrics and started Thinking tool waits at the device end, so the first schedule after the release saw capacity the engine had not freed yet (review finding C-44-3). GlobalBatchEndEvent and the PREFILL branch of ClusterBatchEndEvent now split the output application from the device end. While the host waits at the DP sync, the replica scheduler keeps the application in its batch queue (hold_batch_output); the pass that pops it applies it at the pop time, after that iteration's schedule call, and ReplicaScheduleEvent emits its events. Outputs that end outside the sync are applied at the batch end as before, and the pending samples of preempted steps (C-40-1) go through the same application. The DP placement harness now records each held output's rows, each application and each iteration's schedule call. The new test checks, for an output held in the kv_pressure DP2/PP4 case and its two-round Thinking variant, that nothing changes before the pop, that the lane's first schedule after the hold still sees the rows' KV blocks, and that the output is applied once, at the pop, after that iteration's schedule.
PR 44 grows four modules that were already above the 2,000-line soft limit, and the gate asks for the cleanup done, the reason each touched boundary remains, and a planned split (review finding C-44-7). docs/general/module-size-exceptions.md records the line counts before and after PR 44, the cleanup PR 44 did (kernel-gap and two-stream selection in their own modules, CPU-overhead names and features from the table validator, the duplicated kernel-only attention training removed), why the predictor's measurement-family state keeps the new lookups in the class for now, and a sequenced, movement-only split: CPU-overhead pricing first, then measurement-family binding, with independent ledger and attention-profiler moves. AGENTS.md's size gate links the record.
… the CPU-probe parser Review findings C-44-5 and C-44-6. The parser inferred the KV-load wait as the forward-context entry beyond its median over the stage log, so a slow entry without a KV load lost host work and a KV load common to every step stayed in prepare_inputs_e2e. It counted every one-token request as decode, so a one-token prompt tail was published under the decode identity. The vLLM-BS probe now records the KV connector's start_load_kv stamps per forward and each request's computed tokens and prompt length per step, with the engine's kv_role (vLLM-BS eed75f525). - The KV-load wait is the recorded start_load_kv interval; a forward without a connector subtracts 0. - step_identities counts a request's tokens the way Frontier counts its phase: prefill when they recompute tokens an earlier step computed or continue the prompt, decode otherwise. On a kv_consumer (PD decode) instance a request's first step computes the prompt tokens the connector did not load and is Frontier's first DECODE step. A multi-token decode step (speculative decoding, partial KV load) and an unknown kv_role raise. - Logs without the new fields are rejected instead of guessed. Tests: both review scenarios (a 5 ms entry without a KV load keeps 5.5 ms of prepare; a 5 ms load on every step leaves 0.5 ms), the prompt tail and the decode step publish (1, 1, 0) and (1, 0, 1), the PD decode first step, a resumed preemption victim, and the rejections. 29 passed in the parser file; the related CPU-overhead, DP harness and engine-idle files 34 passed. The simulator does not read this module, so the fidelity matrix is unaffected. Published tables built from older logs need a new GPU recording (X2).
…ler records The padded grouped-GEMM feature used the BLOCK_SIZE_M of the next profiled num_tokens row. vLLM selects the block per step from its default threshold (16 up to the local expert count, 64 above) or the nearest key of a tuned config, so the block can change between two profiled rows. With 128 experts, top-k 2 and rows at 64 and 256 tokens, a 100-token step was padded with 64 (8,192 rows) instead of vLLM's 16 (2,048 rows). The vLLM 0.10.x profiler now evaluates try_get_optimal_moe_config at every token count up to the row's and records the result as time_stats.moe_grouped_gemm.block_size_m_ranges, a JSON list of [first_num_tokens, last_num_tokens, block_size_m]. Training takes the largest row's ranges, rejects a padded table without them, and checks every row's block_size_m against them. The runtime looks the step's token count up in the ranges and still fails above the largest profiled row. Padded tables produced before this change must be re-profiled. No checked-in table has the padded column; the 77-case fidelity matrix is identical.
…sk-record untracking (X1) and the tooling move PR 40 reverted the five calibration tooling commits (4758daf, bca217f, 9577cab, d7e41ee, 6679eb5) so that they reach this branch alone. This branch keeps its current content for every path those commits touch; the conflicts are resolved to it. The merge otherwise removes the tracked task records and the .gitignore exception that published them.
…eview X2) The cloud review asked for the current-table regate to be read and published, for coverage and DP x PP mode boundaries to be classified explicitly, and for a minimum reusable harness before PR 44's numbers are approved. tests/comparison/e2e_gate/ holds one entry point, run_gate.py, with these subcommands: - cells - gate - normalize - frontier - cpu-table (current, k1d or k1e table) - admit Supporting modules: - gate.py: evidence admission (identity, token counts, clock and units, routing status) and the five independent metric decisions under the single, pd_s33 and dp_pp_pairs gate kinds. - normalize.py: builds the normalized rows, reusing e2e_metrics_gap.py and pd_metrics.py. - dp_pp_modes.py: the T43-PAIRGATE mode labels. - admission.py: the I-6, T43-C4PROBE and T43-ISOPREFILL admission rules. cpu-table reruns frontier.profiling.cpu_overhead.vllm_cpu_probe; there is no second publisher. For the k1d and k1e tables it takes --producer-tree at the recorded producer commit, after checking the module's SHA-256. Published data: - cases/scenario_matrix.json: the 22 claimed H200 cells and the excluded C6 qps2, with each cell's coverage class, its gaps, and the members checked. - artifacts.json: Frontier and vLLM-BS identities, ground-truth runs, tables, CPU-overhead recipes and predictor caches, each pinned by SHA-256. - cases/published_readings.json: the X2 reading of every cell at fc94cbc. DP x PP cells add each run's mode windows, the pair counts, and the reference modes that have no gated pairs. Fixtures: - normalized rows of a co-location cell, a PDD cell and a DP x PP cell; - a 100-step CPU-probe slice with its producer CSVs. tests/unit/test_e2e_gate.py replays each fixture against its published entry and covers each admission and gate rule. .gitignore un-ignores the harness JSON and CSV files.
… MATCH Decision C4-ROUTING aligns the C4 reference with Frontier's balanced MoE split: the C4 clean ground truth now runs with vLLM-BS round-robin routing (moe_uniform_routing), as C5, C6 and C2' do. H200 routing-records runs give a per-expert count error of 0 for C4 (both PD instances) and for C2', so both cases leave UNSET and are gated. - cases/scenario_matrix.json: C4 cells point at the c4-rr-h200-clean-*-20261011 runs; C4 and C2' routing status MATCH; manifest hashes updated. - artifacts.json: pins of the new C4 ground-truth runs. - cases/published_readings.json: 16 PASS, 6 FAIL, 0 INSUFFICIENT_EVIDENCE. C4 qps37 passes; C4 qps2, qps75 and qps37_mixed_warm fail on TTFT; C2' qps2/qps17 pass and qps8/qps34 fail. The Frontier runs are unchanged. - README.md: the routing limit now states the round-robin reference.
…ion PD-POOL) The C4 TTFT errors moved across the 0.10 limit depending on which single same-commit reference run the matrix named (host speed on the vLLM side). A PDD cell now lists every clean run that passed its run check and meets the case's routing status, and the gate pools them: - gate.py: single_reading sums each metric's numerator and denominator over the cell's vLLM runs, so a latency reference is the mean of the runs' means and a throughput reference is the pooled formal requests or tokens over the summed formal windows. One run gives its own value unchanged. For pd_s33 each run's TTFT is reduced by its own minimum endpoint offset; ttft_s33 records every run and the pooled adjusted value. - Matrix: C3 qps2 pools 2 runs and qps37_p512 5 runs; each C4 cell pools its round-robin run of 2026-10-11 and two new ones of job exp-1012-135101-963575 (decision C4-ROUTING). The first listed run, which sets Frontier's KV block count, is unchanged, so every Frontier command is unchanged. - Readings: 17 PASS / 5 FAIL (was 16 / 6). C3 qps37_p512 FAIL 0.122 -> PASS 0.080. C4 qps2, qps75 and qps37_mixed_warm still FAIL (TTFT 0.112, 0.249, 0.138); qps37 PASS 0.090. Co-location cells keep their one named run. - The c3_qps2 fixture now carries both pooled vLLM runs; a unit test covers pooling with per-run S33 bounds.
…e (decision POOL-COLOC) A co-location single cell now lists every clean vLLM run that passed its run check and meets the case's routing status, as PDD cells do under PD-POOL. In the eager MoE co-location case C5 a run must also come from a job that first ran an unmeasured warm pass over a trace with the cell's prompt lengths (T43-COMPILECACHE). The C5 qps2 base-job run, which T43-COMPILECACHE did not rerun, stays listed. Excluded: the C5 cell job of 2026-10-07 (no warm pass) and c5-rr-h200-clean-qps2_mixed_warm-20261010c, whose job warmed only the 2048-token qps2 trace (TTFT 215.934 ms against 122.996-124.704 ms in the admissible runs). The first listed run of each cell is unchanged, so the recorded Frontier commands and KV block counts are unchanged. Pooled vLLM TTFT against Frontier (all six cells still PASS): - C1 qps2: 2 runs, 51.785 vs 51.097 ms (0.0133) - C1 qps12.5: 2 runs, 614.420 vs 635.106 ms (0.0337) - C1 qps6.5_mixed_warm: 3 runs, 72.582 vs 72.909 ms (0.0045) - C5 qps2: 2 runs, 112.774 vs 114.776 ms (0.0177) - C5 qps5: 3 runs, 129.719 vs 124.912 ms (0.0371) - C5 qps2_mixed_warm: 5 runs, 124.005 vs 119.288 ms (0.0380) Files: README gate-kind text and fixture line; scenario_matrix.json (groundtruth_runs of the six cells only); artifacts.json (11 run records added); published_readings.json (gated entries of the six cells only); the c1_qps2 fixture rows with both C1 qps2 runs. 26 unit tests pass; the c1_qps2, c3_qps2 and c6_qps0.75_drain fixture replays reproduce the published verdicts.
…ecision C4-DRAINFLOOR) ExecutionTime.forward_launch_stall_time now returns max(0, launch + drain - device span) instead of charging launch + drain only when the launch outlasts the device span (T43-PDHOST option D -> A). The CPU table measures both terms in situ on the probe node. On H200 device-bound prefills the kernel-only device span is short of the measured device time (C4 2304-3583 tokens by 2.067-2.673 ms; the dense connector engine C3 by 2.306 ms, 46.119 against 43.813 ms), while launch + drain reproduces the probes (13.955 against 13.988 ms, 16.010 against 15.997 ms). Under option D those steps dropped the drain and priced the shorter device span. A table without forward_launch/forward_drain columns gives 0 for both, so the rule leaves the public examples and older tables unchanged: the 77-case refactor fidelity matrix is identical, and the unit + integration failure ids equal the baseline (107). The new parametrized test covers the launch-bound, device-bound-below-floor, device-bound-above-floor and no-table cases; the device-bound-below-floor case fails on the previous rule.
…cision C4-DRAINFLOOR) All 50 Frontier runs of the 22 claimed cells (15 single-run cells, 35 DP x PP members) were rerun at 5a8b86c with the X2 settings, matrix, predictor cache and route-delay seeds. Each run's metrics files are byte-identical to the in-process drain-floor what-if of the C4-CPU-RCA (config.json up to paths). - scenario_matrix.json: every Frontier run's frontier_revision is 5a8b86c. Coverage is rechecked on the new runs. C6 qps3.5_drain becomes outside_profile: in member s5, 298 formal batches reach batch_size / num_decode_tokens 51 against a profiled maximum of 50 in the CPU table (3241.696 ms). The C1 and C5 mixed cells stay outside_profile: 3 batches / 273.503 ms and 2 / 87.083 ms. - artifacts.json: Frontier commit 5a8b86c. The current CPU-overhead tables keep their producer commit fc94cbc (cpu_overhead_producer_commit); frontier/profiling/cpu_overhead is unchanged. - published_readings.json: 17 PASS / 5 FAIL. C4 qps37_mixed_warm turns PASS (TTFT 19.165 against vLLM 20.666 ms, 0.0726). C6 qps3.5_drain turns FAIL: its members now run in split mode, where TTFT reads 484.904 against 424.808 ms (0.1415). - Fixtures: the Frontier rows of c1_qps2 (TTFT 51.286 against 51.785 ms), c3_qps2 (49.216 against 49.897 ms) and c6_qps0.75_drain (in_phase TTFT 305.300 against 305.810 ms; 42 windows per member). Checks: tests/unit/test_e2e_gate.py 26 passed; the three fixture replays reproduce their entries; cpu-table --table current rebuilds all six cases' tables to the pinned SHA-256; the harness readings agree with gate_cells.py or the per-run mean within 2.8e-16.
… tables In a DP x EP instance the engines meet in every layer's expert-parallel all-to-all, so a forward that runs while another DP engine runs a prompt lasts until that engine's device work ends. The producer folded that wait into the forward_launch and forward_drain medians of whichever shapes were sampled mostly beside a peer prompt (C6 prefill rows: 122-142 ms of launch + drain against a 94-100 ms span without a peer prompt). Frontier already simulates the coupling between DP lanes, and the drain floor (5a8b86c) then imposed the coupled time a second time on every step of those shapes, which lengthened split-mode prefills by 10-17 ms per stage and failed C6 qps3.5_drain (split TTFT 484.904 against vLLM 424.808 ms). The producer now leaves out of the rows a forward whose device interval overlaps a same-stage forward of another DP engine that holds prefill tokens. forward_device_intervals holds the launch-start / device-end chain that stage_overhead_terms already used. DP=1 instances are unchanged: the C1, C3, C4 and C5 tables rebuild byte for byte; the C6 and C2' tables lose 730 and 298 of 9508 and 27762 forwards. Checks: producer tests 30 passed (the new test fails on 5a8b86c); unit + integration suite failure set unchanged (107 ids); refactor fidelity matrix 77/77 identical; with the rebuilt table C6 qps3.5_drain reads split TTFT 440.003 against 424.808 ms (0.0358).
(decision REMAINING-FAIL) c0abc05 leaves the forwards of a DP>1 instance that ran beside another DP engine's prompt out of the CPU-probe tables. The X3 tables are the X2 tables rebuilt at c0abc05 from the same probe runs: C1, C3, C4 and C5 are byte-identical, C6 and C2' change. All 50 Frontier runs of the 22 claimed cells were rerun at c0abc05 with the X3 settings (the X2 settings with the X3 tables) and X2's matrix and route-delay seeds. The 45 runs with an earlier run on the same table are byte-identical to it (config.json up to paths); the five C6 qps0.75_drain members have none. - scenario_matrix.json: run labels x3 / x3_s<seed>, frontier_revision c0abc05, the X3 settings and table paths. Coverage is rechecked: C6 qps3.5_drain is in_profile again (member s5 no longer reaches the CPU table's batch-size edge); the C1 and C5 mixed cells keep their gaps. - artifacts.json: Frontier commit and CPU producer commit c0abc05; current CPU-overhead recipe cpu_overhead_h200_x3; predictor cache of the X3 settings (only the CPU-overhead models differ). - published_readings.json: 18 PASS / 4 FAIL. - C6 qps3.5_drain FAIL -> PASS: split TTFT 440.003 against vLLM 424.808 ms (0.0358; was 484.904 ms). - C2' qps34 FAIL -> PASS: in_phase TPOT 26.604 against 25.402 ms (0.0473), TTFT 39.069 against 39.647 ms (0.0146). - C2' qps17 PASS -> FAIL: member s2 now runs in split, where TPOT reads 36.389 against 31.849 ms (0.1425). - C6 qps2_drain stays PASS: in_phase TTFT 306.482 against 313.606 ms (0.0227; was 330.256 ms). - C6 qps0.75_drain stays PASS: in_phase TTFT 295.777 against 305.810 ms (0.0328; was 305.300 ms), TPOT 188.238 against 197.320 ms (0.0460). - C4 qps2, C4 qps75 and C2' qps8 stay FAIL (C2' qps8 split TPOT 31.918 against 28.422 ms). - Fixtures: the Frontier rows are renamed to the x3 labels. c1_qps2 and c3_qps2 keep their content; c6_qps0.75_drain carries the X3 rows (41 in_phase windows per member). - run_gate.py and README.md name the current table X3. Checks: tests/unit/test_e2e_gate.py 26 passed; the three fixture replays reproduce their entries; cpu-table --table current rebuilds all six cases' tables to the pinned SHA-256; the harness readings agree with gate_cells.py or the per-run mean within 2.8e-16.
…ion KV-SEND-CORRECTION) vLLM's P2pNcclConnector sends each layer's KV of the prompts a prefill step completes while the layer's later operators run, and those operators slow down (H800 op timing, 2048 tokens: +3.983 ms dense, +2.476 ms MoE per forward; H200: the C3 forward runs 46.119 ms on the device against Frontier's 43.813 ms). Frontier does not model the send, so its priced device span misses this time. - kv_cache_transfer_config.kv_send_slowdown_per_layer_ms (default 0.0) adds that device time to each layer of a PDD prefill step that completes at least one prompt, once per layer however many prompts the step sends. The C3 H200 probes show a per-step, not per-prompt, cost: steps that complete 1 to 5 prompts run 0.254 to 4.228 ms longer than Frontier's span. - The time joins the connector's KV extract term of the layer, so the forward composition max(S, L + D) takes it without double counting what an in-situ CPU table already carries. - A value needs a kv_connector; runs without one keep their outputs (77/77 public-example cases identical apart from the new config.json field). This is a temporary measured correction; a model of the layer-wise send is deferred.
…n (decision KV-SEND-CORRECTION) All 50 gate-cell runs rerun at 222e4e2. C3 and C4 use the X4 settings (the X3 settings plus kv_send_slowdown_per_layer_ms: 0.0721 ms per layer for C3, 0.101 for C4); the other four cases keep their X3 settings and their runs are byte-identical to the X3 ones apart from the new config.json field. 18 PASS / 4 FAIL, the same verdicts as 89c0e2a. Changes, vLLM / before / after: - C4 qps37 TTFT 13.197 / 12.008 / 12.294 ms (0.0901 -> 0.0684). - C4 qps75 TTFT 21.289 / 16.234 / 16.861 ms (0.2374 -> 0.2080), TPOT 3.537 / 3.154 / 3.141 ms, E2E 942.037 / 825.594 / 822.781 ms; still FAIL. - C4 qps2 TTFT 13.120 / 11.653 / 11.674 ms; still FAIL. - C3 qps2 TTFT 49.897 / 49.216 / 49.292 ms; qps8 TTFT 55.466 / 54.001 / 54.023 ms; qps37_p512 unchanged. The CPU-overhead tables and their producer commit (c0abc05) are unchanged. Coverage is identical to the X3 round: the same two GAP lines on the C1 and C5 mixed cells, and C3 qps8 reports 3 more cpu_exact_key_misses (32 -> 35) from the shifted batching.
…nd correction Commit 222e4e2 added the measured KV-send slowdown term to sklearn_execution_time_predictor.py, which grew it from 8,441 to 8,464 lines. The module-size exception record (review finding C-44-7) still listed 8,441 and did not name the term. The other three inventory rows already match the current files.
fwyc0573
changed the base branch from
fix/vllm-v1-skipped-requeue-order
to
main
October 14, 2026 12:33
fwyc0573
marked this pull request as ready for review
October 14, 2026 12:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
The H800 and H200 numeric calibration of Frontier against measured vLLM 0.10.2 ground truth. This is
the RCA tail of the end-to-end accuracy gate requested on 2026-09-27, carried as tickets 12-43 of that
gate. Stacked on #42; its base is
fix/vllm-v1-skipped-requeue-order, so the diff shown is thecalibration work only.
This PR deliberately changes numeric results. It is the opposite of the #34 -> #42 stack, whose
claim is that semantics change and numbers do not. Keep the two apart when reading a moved number.
frontier/tests/63 non-merge commits. Ten local topic branches are consolidated here by merge, never by rebase, because
the recorded gate readings pin individual commits by
frontier_revision:calibration/i4-native-op-timing,calibration/eager-device-stream,calibration/t38-stage-cpu-overhead,calibration/t40-dp-dummy-pass,t43-rca-globalpad,calibration/t41-unified-cpu-overhead,calibration/t43-pd-host-terms,calibration/t43-dp-pp-modes,calibration/stack-e2e-gate,calibration/t43-combined.Result
Gate: H200 +
step_main, six topologies, 22 cells, every metric gated independently at relative error<= 0.10, all online with a profiled (non-dummy) predictor.
21 of 22 cells PASS. Starting point on 2026-09-28, before this chain: C1 eager TPOT 0.3161, C2 TPOT
0.8519, C3 TTFT 0.2016, C4 TTFT 0.4755, C5 TPOT 0.7330.
The fidelity changes
Each one is an approved decision with measured evidence; each moves numbers on purpose.
3ed37deb0bf0c7c7aabee,7df94b72c9d625ced720b,58750e6f1091cd9c104bc0fb8d7e,a3cead0476b69ec622d011908f2f27fc9b62e97873b65ddbe0f5a8876ec5e75applied then reverted6ec5e75,3a1add0,4c0aee7Measurement and tooling, the rest of the diff: the kernel-gap profiler and its graph modes, the
grouped-GEMM
BLOCK_SIZE_Mand padded-token features, the CPU-overhead CSV builder from vLLMCPU-probe logs, the 1P1D ground-truth harness, the NCCL point-to-point benchmark, the TP all-reduce
timing modes, the shared E2E gate tool, and the replay harness changes
(
tests/comparison/dp_placement_pp/).Known gaps
rule found no paired request in the frequent vLLM mode. The cause is in the reference — the job's
five runs each settled in a different DP x PP lock mode, and the clean runs' own TPOT medians span
1.82x (39.02, 25.70, 21.42 ms). Where both sides visit the same mode, Frontier's per-token
period matches vLLM's (I-28). More simulator changes cannot close this; a multi-modal reference rule
or a declared boundary can.
tables. Issue I-36 then found the probe-admission check of T41-I6MODE rejected 3 of 5 C2' probes for
reasons independent of host speed, and T43-ISOPREFILL replaced it; the k1e tables were published
and the re-gate was started but has not been read.
tables.
measured profiling grids.
attn_dp > 1in dense or PDD, named open by ticket 42.attn_dp > 1is covered only in MoE co-location (C2', C6).
the CPU-overhead publisher are about 17,675 lines across 108 Python files under an ignored
task_memory/path. A reviewer cannot re-derive the numbers above from this diff alone. Migratingthe reusable part (roughly 4,900 lines in 25 files) into
tests/comparison/e2e_gate/is a separate,inventoried change.
Not in this PR
verdict: across 30 metric cells of five cases at the stack head and at
origin/main, no cell movesbetween PASS and FAIL, and three of five cases are bit-identical. The accuracy gaps this PR closes
were present on
main.Mode have no gate cell. Neither does PD-AF, and neither does any offline cell. All six cases are
online.