From afe37cbe601c4849d2f8d3d0e4222859c9eefa55 Mon Sep 17 00:00:00 2001 From: Chang Liu <9713593+chang-l@users.noreply.github.com> Date: Mon, 13 Apr 2026 17:00:13 -0700 Subject: [PATCH 1/2] [https://nvbugs/6050483][fix] Pin diffusers to 0.37.1 to fix UniPC scheduler device mismatch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The VBench integration tests install diffusers from git HEAD, which pulled in huggingface/diffusers#13356. That PR changed `UniPCMultistepScheduler.multistep_uni_p_bh_update` to use `torch.stack(rks)` on a list mixing CPU tensors (from `self.sigmas`, intentionally kept on CPU) with a CUDA tensor (`torch.ones((), device=device)`), causing `RuntimeError: Expected all tensors to be on the same device`. Pin diffusers to 0.37.1 (latest stable release) which uses the original correct pattern: `torch.tensor(rks, device=device)`. Verified on B200: - diffusers 0.38.0.dev0 → crashes at UniPC step 1 - diffusers 0.37.1 → passes all steps Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com> --- tests/integration/defs/examples/test_visual_gen.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/integration/defs/examples/test_visual_gen.py b/tests/integration/defs/examples/test_visual_gen.py index 77b80d805779..31178522243b 100644 --- a/tests/integration/defs/examples/test_visual_gen.py +++ b/tests/integration/defs/examples/test_visual_gen.py @@ -159,7 +159,10 @@ def _visual_gen_deps(llm_venv): """Install av + diffusers + ffmpeg once per session (shared by all video-gen fixtures).""" llm_venv.run_cmd(["-m", "pip", "install", "av"]) - llm_venv.run_cmd(["-m", "pip", "install", "git+https://github.com/huggingface/diffusers.git"]) + # Pin diffusers to 0.37.1 to avoid device-mismatch regression in + # UniPCMultistepScheduler.multistep_uni_p_bh_update introduced by + # huggingface/diffusers#13356 (torch.stack on mixed CPU/CUDA rks). + llm_venv.run_cmd(["-m", "pip", "install", "diffusers==0.37.1"]) # Install ffmpeg system package required by MediaStorage.save_video for MP4 encoding check_call(["apt-get", "update", "-y"], shell=False) check_call(["apt-get", "install", "-y", "ffmpeg"], shell=False) From dc7ae45e098902452c1756ed81f9338c21b39352 Mon Sep 17 00:00:00 2001 From: Chang Liu <9713593+chang-l@users.noreply.github.com> Date: Tue, 14 Apr 2026 09:18:37 -0700 Subject: [PATCH 2/2] [https://nvbugs/6050483][fix] Pin diffusers to 0.37.1 and remove test waivers Remove 5 test waivers added for nvbug 6050483 so CI will exercise the previously-failing visual gen tests with the pinned diffusers version. Removed waivers: - examples/test_visual_gen.py::test_vbench_dimension_score_wan22_a14b_nvfp4 - examples/test_visual_gen.py::test_vbench_dimension_score_wan22_a14b_fp8 - examples/test_visual_gen.py::test_vbench_dimension_score_wan - visual_gen/test_visual_gen_benchmark.py::test_online_benchmark[openai-videos] - visual_gen/test_visual_gen_benchmark.py::test_offline_benchmark Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com> --- tests/integration/test_lists/waives.txt | 5 ----- 1 file changed, 5 deletions(-) diff --git a/tests/integration/test_lists/waives.txt b/tests/integration/test_lists/waives.txt index 5a963d90f818..e6f39617979b 100644 --- a/tests/integration/test_lists/waives.txt +++ b/tests/integration/test_lists/waives.txt @@ -317,12 +317,7 @@ perf/test_perf.py::test_perf[deepseek_r1_distill_qwen_32b-bench-_autodeploy-floa accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v1_kv_cache-dp4-triton-auto] SKIP (https://nvbugs/6026678) accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v1_kv_cache-ep4-triton-auto] SKIP (https://nvbugs/6026678) accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-ep4-trtllm-fp8] SKIP (https://nvbugs/6050481) -examples/test_visual_gen.py::test_vbench_dimension_score_wan22_a14b_nvfp4 SKIP (https://nvbugs/6050483) -visual_gen/test_visual_gen_benchmark.py::test_online_benchmark[openai-videos] SKIP (https://nvbugs/6050483) accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v1_kv_cache-ep4-trtllm-fp8] SKIP (https://nvbugs/6050481) -examples/test_visual_gen.py::test_vbench_dimension_score_wan SKIP (https://nvbugs/6050483) -examples/test_visual_gen.py::test_vbench_dimension_score_wan22_a14b_fp8 SKIP (https://nvbugs/6050483) -visual_gen/test_visual_gen_benchmark.py::test_offline_benchmark SKIP (https://nvbugs/6050483) accuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_bfloat16_4gpus[pp4-attn_backend=TRTLLM-torch_compile=False] SKIP (https://nvbugs/6050487) accuracy/test_llm_api_pytorch.py::TestDeepSeekV3Lite::test_bfloat16_4gpus[pp4-mtp_nextn=0-attention_dp=False-cuda_graph=True-overlap_scheduler=False-torch_compile=False] SKIP (https://nvbugs/6050489) disaggregated/test_disaggregated.py::test_disaggregated_gpt_oss_120b_harmony[gpt_oss/gpt-oss-120b] SKIP (https://nvbugs/6011317)