stable_diffusion_3 model/pipeline review

Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423

Review performed against the repository review rules.

Issue 1: SD3 inpaint decode drops the VAE shift

Affected code:

Problem:
StableDiffusion3InpaintPipeline encodes VAE latents with (latents - shift_factor) * scaling_factor, but decodes with only latents / scaling_factor. The missing + self.vae.config.shift_factor makes inpaint decoding inconsistent with the other SD3 pipelines and with its own encode path. Duplicate search found no matching issue/PR.

Impact:
User-visible inpaint outputs are decoded from the wrong latent distribution whenever the VAE has a nonzero shift_factor, which SD3 VAEs do.

Reproduction:

import torch
from diffusers import AutoencoderKL

vae = AutoencoderKL(
    sample_size=8, in_channels=3, out_channels=3, block_out_channels=(4,),
    layers_per_block=1, latent_channels=16, norm_num_groups=1,
    use_quant_conv=False, use_post_quant_conv=False,
    shift_factor=0.25, scaling_factor=2.0,
)
latents = torch.randn(1, 16, 8, 8)
current = vae.decode(latents / vae.config.scaling_factor, return_dict=False)[0]
expected = vae.decode((latents / vae.config.scaling_factor) + vae.config.shift_factor, return_dict=False)[0]
print((current - expected).abs().max())

Relevant precedent:

Suggested fix:

latents = (latents / self.vae.config.scaling_factor) + self.vae.config.shift_factor
image = self.vae.decode(latents, return_dict=False, generator=generator)[0]

Issue 2: SD3 ControlNet pipelines cannot use dynamic-shifting schedulers

Affected code:

Problem:
The base SD3 pipelines compute and pass mu when FlowMatchEulerDiscreteScheduler.config.use_dynamic_shifting=True. The ControlNet SD3 pipelines call retrieve_timesteps() without mu handling and expose no mu argument. Duplicate search found no matching issue/PR.

Impact:
SD3.5-style scheduler configs with dynamic shifting fail before inference, so ControlNet is inconsistent with the rest of the SD3 family.

Reproduction:

from diffusers import FlowMatchEulerDiscreteScheduler
from diffusers.pipelines.controlnet_sd3.pipeline_stable_diffusion_3_controlnet import retrieve_timesteps

scheduler = FlowMatchEulerDiscreteScheduler(use_dynamic_shifting=True)
retrieve_timesteps(scheduler, num_inference_steps=2, device="cpu")

Relevant precedent:

Suggested fix:
Add the same mu argument, calculate_shift() logic, and scheduler_kwargs["mu"] handling used by the base SD3 pipelines before calling retrieve_timesteps().

Issue 3: Duplicate: controlnet_pooled_projections tensor path is broken

Affected code:

Problem:
This is already reported in open issue #9686. When a user passes a tensor for controlnet_pooled_projections, the code evaluates it with Python or, which raises RuntimeError: Boolean value of Tensor with more than one value is ambiguous.

Impact:
The public controlnet_pooled_projections argument cannot be used reliably, and SD3 ControlNet inference/training validation can diverge from the intended pooled-projection conditioning path.

Reproduction:

import torch

controlnet_pooled_projections = torch.ones(1, 8)
pooled_prompt_embeds = torch.zeros(1, 8)
controlnet_pooled_projections = controlnet_pooled_projections or pooled_prompt_embeds

Relevant precedent:
Existing duplicate: #9686

Suggested fix:

if controlnet_config.force_zeros_for_pooled_projection:
    controlnet_pooled_projections = torch.zeros_like(pooled_prompt_embeds)
elif controlnet_pooled_projections is None:
    controlnet_pooled_projections = pooled_prompt_embeds

Issue 4: SD3ControlNetModel.from_transformer() mutates the source transformer config

Affected code:

Problem:
config = transformer.config aliases the transformer's live config, then writes ControlNet-specific values into it. Duplicate search found no matching issue/PR.

Impact:
Calling from_transformer() silently changes transformer.config.num_layers and adds extra_conditioning_channels, which can corrupt later serialization, logging, or pipeline construction using the original transformer.

Reproduction:

from diffusers import SD3ControlNetModel, SD3Transformer2DModel

transformer = SD3Transformer2DModel(
    sample_size=4, patch_size=1, in_channels=4, out_channels=4, num_layers=3,
    attention_head_dim=4, num_attention_heads=2, caption_projection_dim=8,
    joint_attention_dim=8, pooled_projection_dim=8,
)
print(dict(transformer.config).get("num_layers"))
SD3ControlNetModel.from_transformer(transformer, num_layers=1, num_extra_conditioning_channels=2, load_weights_from_transformer=False)
print(dict(transformer.config).get("num_layers"), dict(transformer.config).get("extra_conditioning_channels"))

Relevant precedent:

Suggested fix:

config = dict(transformer.config)
config["num_layers"] = num_layers or transformer.config.num_layers
config["extra_conditioning_channels"] = num_extra_conditioning_channels
controlnet = cls.from_config(config)

Issue 5: ControlNet inpaint rejects documented IP-Adapter image embeds

Affected code:

Problem:
The docstring and prepare_ip_adapter_image_embeds() path expect ip_adapter_image_embeds to be a tensor, but check_inputs() rejects tensors and requires a list. Duplicate search found no matching issue/PR.

Impact:
Users cannot pass precomputed IP-Adapter image embeddings to StableDiffusion3ControlNetInpaintingPipeline even though the public signature documents that path.

Reproduction:

import torch
from diffusers import StableDiffusion3ControlNetInpaintingPipeline

pipe = object.__new__(StableDiffusion3ControlNetInpaintingPipeline)
pipe.vae_scale_factor = 1
pipe.patch_size = 1
pipe._callback_tensor_inputs = ["latents"]
pipe.controlnet = object()
pipe.check_inputs(
    height=8, width=8, image=torch.zeros(1, 3, 8, 8),
    prompt=None, prompt_2=None, prompt_3=None,
    prompt_embeds=torch.zeros(1, 2, 8),
    pooled_prompt_embeds=torch.zeros(1, 8),
    ip_adapter_image_embeds=torch.zeros(1, 2, 8),
    control_guidance_start=[0.0], control_guidance_end=[1.0],
)

Relevant precedent:

Suggested fix:

if ip_adapter_image_embeds is not None and ip_adapter_image_embeds.ndim not in [3, 4]:
    raise ValueError(
        f"`ip_adapter_image_embeds` has to be a 3D or 4D tensor but is {ip_adapter_image_embeds.ndim}D"
    )

Issue 6: Slow tests are missing for SD3 inpaint and SD3 ControlNet inpaint

Affected code:

Problem:
Both files only define fast tests. The target family has slow coverage for SD3 text-to-image, SD3 img2img, and SD3 ControlNet, but not for the two inpaint variants. Duplicate search found no matching issue/PR.

Impact:
Real checkpoint behavior, offload behavior, VAE shift handling, and image/mask preprocessing are not covered for the inpaint variants. This gap would have allowed Issue 1 to remain invisible in CI.

Reproduction:

from pathlib import Path

for path in [
    Path("tests/pipelines/stable_diffusion_3/test_pipeline_stable_diffusion_3_inpaint.py"),
    Path("tests/pipelines/controlnet_sd3/test_controlnet_inpaint_sd3.py"),
]:
    text = path.read_text()
    print(path, "@slow" in text, "SlowTests" in text)

Relevant precedent:

Suggested fix:
Add @slow / @require_big_accelerator classes for StableDiffusion3InpaintPipeline and StableDiffusion3ControlNetInpaintingPipeline using real SD3-family checkpoints, CPU/GPU offload, and deterministic output-slice assertions.