Repository navigation
controlnet model/pipeline review #13600
Description
controlnet model/pipeline review
Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423
Review performed against the repository review rules.
Issue 1: MultiControlNetUnionModel is missing from top-level exports
Affected code:
diffusers/src/diffusers/__init__.py
Lines 229 to 269 in 0f1abc4
| "ConsistencyDecoderVAE", | |
| "ContextParallelConfig", | |
| "ControlNetModel", | |
| "ControlNetUnionModel", | |
| "ControlNetXSAdapter", | |
| "CosmosControlNetModel", | |
| "CosmosTransformer3DModel", | |
| "DiTTransformer2DModel", | |
| "EasyAnimateTransformer3DModel", | |
| "ErnieImageTransformer2DModel", | |
| "Flux2Transformer2DModel", | |
| "FluxControlNetModel", | |
| "FluxMultiControlNetModel", | |
| "FluxTransformer2DModel", | |
| "GlmImageTransformer2DModel", | |
| "HeliosTransformer3DModel", | |
| "HiDreamImageTransformer2DModel", | |
| "HunyuanDiT2DControlNetModel", | |
| "HunyuanDiT2DModel", | |
| "HunyuanDiT2DMultiControlNetModel", | |
| "HunyuanImageTransformer2DModel", | |
| "HunyuanVideo15Transformer3DModel", | |
| "HunyuanVideoFramepackTransformer3DModel", | |
| "HunyuanVideoTransformer3DModel", | |
| "I2VGenXLUNet", | |
| "Kandinsky3UNet", | |
| "Kandinsky5Transformer3DModel", | |
| "LatteTransformer3DModel", | |
| "LongCatAudioDiTTransformer", | |
| "LongCatAudioDiTVae", | |
| "LongCatImageTransformer2DModel", | |
| "LTX2VideoTransformer3DModel", | |
| "LTXVideoTransformer3DModel", | |
| "Lumina2Transformer2DModel", | |
| "LuminaNextDiT2DModel", | |
| "MochiTransformer3DModel", | |
| "ModelMixin", | |
| "MotionAdapter", | |
| "MultiAdapter", | |
| "MultiControlNetModel", | |
| "NucleusMoEImageTransformer2DModel", |
diffusers/src/diffusers/models/__init__.py
Lines 75 to 79 in 0f1abc4
| _import_structure["controlnets.controlnet_union"] = ["ControlNetUnionModel"] | |
| _import_structure["controlnets.controlnet_xs"] = ["ControlNetXSAdapter", "UNetControlNetXSModel"] | |
| _import_structure["controlnets.controlnet_z_image"] = ["ZImageControlNetModel"] | |
| _import_structure["controlnets.multicontrolnet"] = ["MultiControlNetModel"] | |
| _import_structure["controlnets.multicontrolnet_union"] = ["MultiControlNetUnionModel"] |
Problem:
MultiControlNetUnionModel is exported from diffusers.models, but not from diffusers, while adjacent public ControlNet classes are top-level exports.
Impact:
Users cannot follow the standard from diffusers import ... pattern for this public wrapper.
Reproduction:
from diffusers import ControlNetUnionModel, MultiControlNetModel
print(ControlNetUnionModel, MultiControlNetModel)
from diffusers import MultiControlNetUnionModelRelevant precedent:
MultiControlNetModel is top-level exported.
Suggested fix:
# src/diffusers/__init__.py
# Add "MultiControlNetUnionModel" next to "MultiControlNetModel"
# in both the lazy _import_structure["models"] list and TYPE_CHECKING imports.Issue 2: ControlNetUnionModel() default constructor crashes
Affected code:
| addition_time_embed_dim: int | None = None, |
diffusers/src/diffusers/models/controlnets/controlnet_union.py
Lines 307 to 308 in 0f1abc4
| self.control_type_proj = Timesteps(addition_time_embed_dim, flip_sin_to_cos, freq_shift) | |
| self.control_add_embedding = TimestepEmbedding(addition_time_embed_dim * num_control_type, time_embed_dim) |
Problem:
addition_time_embed_dim defaults to None, but the constructor always uses it to create Timesteps and TimestepEmbedding.
Impact:
A documented public model constructor fails before forward or serialization can be tested.
Reproduction:
from diffusers import ControlNetUnionModel
ControlNetUnionModel(
in_channels=4,
conditioning_channels=3,
down_block_types=("DownBlock2D",),
block_out_channels=(8,),
layers_per_block=1,
norm_num_groups=4,
cross_attention_dim=16,
attention_head_dim=1,
num_trans_channel=8,
num_trans_head=1,
num_proj_channel=8,
conditioning_embedding_out_channels=(4, 8),
)Relevant precedent:
ControlNetModel only constructs add_time_proj when the matching addition embedding mode requires it.
Suggested fix:
if addition_time_embed_dim is None:
raise ValueError("`addition_time_embed_dim` must be set for `ControlNetUnionModel`.")Issue 3: Union pipelines advertise control_mode=None but crash
Affected code:
diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py
Line 1007 in 0f1abc4
| control_mode: int | list[int] | list[list[int]] | None = None, |
diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py
Lines 1184 to 1189 in 0f1abc4
| if not isinstance(control_mode, list): | |
| control_mode = [control_mode] | |
| if isinstance(controlnet, MultiControlNetUnionModel): | |
| control_image = [[item] for item in control_image] | |
| control_mode = [[item] for item in control_mode] |
diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py
Lines 793 to 805 in 0f1abc4
| # Check `control_mode` | |
| if isinstance(controlnet, ControlNetUnionModel): | |
| if max(control_mode) >= controlnet.config.num_control_type: | |
| raise ValueError(f"control_mode: must be lower than {controlnet.config.num_control_type}.") | |
| elif isinstance(controlnet, MultiControlNetUnionModel): | |
| for _control_mode, _controlnet in zip(control_mode, self.controlnet.nets): | |
| if max(_control_mode) >= _controlnet.config.num_control_type: | |
| raise ValueError(f"control_mode: must be lower than {_controlnet.config.num_control_type}.") | |
| # Equal number of `image` and `control_mode` elements | |
| if isinstance(controlnet, ControlNetUnionModel): | |
| if len(image) != len(control_mode): | |
| raise ValueError("Expected len(control_image) == len(control_mode)") |
Problem:
The Union pipelines default control_mode to None, wrap it into [None], then compare None >= num_control_type.
Impact:
Calling a Union pipeline without explicitly passing control_mode fails with a TypeError.
Reproduction:
from types import MethodType
from diffusers import ControlNetUnionModel, StableDiffusionXLControlNetUnionPipeline
pipe = object.__new__(StableDiffusionXLControlNetUnionPipeline)
pipe._callback_tensor_inputs = []
pipe.check_image = MethodType(lambda self, image, prompt, prompt_embeds: None, pipe)
pipe.controlnet = ControlNetUnionModel(
in_channels=4,
conditioning_channels=3,
down_block_types=("DownBlock2D",),
block_out_channels=(8,),
layers_per_block=1,
norm_num_groups=4,
cross_attention_dim=16,
attention_head_dim=1,
addition_time_embed_dim=8,
num_trans_channel=8,
num_trans_head=1,
num_proj_channel=8,
conditioning_embedding_out_channels=(4, 8),
)
control_mode = None
if not isinstance(control_mode, list):
control_mode = [control_mode]
pipe.check_inputs(
prompt="a prompt",
prompt_2=None,
image=[object()],
control_guidance_start=[0.0],
control_guidance_end=[1.0],
control_mode=control_mode,
callback_on_step_end_tensor_inputs=[],
)Relevant precedent:
Merged PR #10747 added multi-union handling but did not make None a valid default.
Suggested fix:
if control_mode is None:
control_mode = 0
if not isinstance(control_mode, list):
control_mode = [control_mode]Issue 4: Multi-ControlNet scale length validation is unreachable or missing
Affected code:
diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet.py
Lines 701 to 712 in 0f1abc4
| if isinstance(controlnet_conditioning_scale, list): | |
| if any(isinstance(i, list) for i in controlnet_conditioning_scale): | |
| raise ValueError( | |
| "A single batch of varying conditioning scale settings (e.g. [[1.0, 0.5], [0.2, 0.8]]) is not supported at the moment. " | |
| "The conditioning scale must be fixed across the batch." | |
| ) | |
| elif isinstance(controlnet_conditioning_scale, list) and len(controlnet_conditioning_scale) != len( | |
| self.controlnet.nets | |
| ): | |
| raise ValueError( | |
| "For multiple controlnets: When `controlnet_conditioning_scale` is specified as `list`, it must have" | |
| " the same length as the number of controlnets" |
diffusers/src/diffusers/models/controlnets/multicontrolnet.py
Lines 47 to 63 in 0f1abc4
| for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, self.nets)): | |
| down_samples, mid_sample = controlnet( | |
| sample=sample, | |
| timestep=timestep, | |
| encoder_hidden_states=encoder_hidden_states, | |
| controlnet_cond=image, | |
| conditioning_scale=scale, | |
| class_labels=class_labels, | |
| timestep_cond=timestep_cond, | |
| attention_mask=attention_mask, | |
| added_cond_kwargs=added_cond_kwargs, | |
| cross_attention_kwargs=cross_attention_kwargs, | |
| guess_mode=guess_mode, | |
| return_dict=return_dict, | |
| ) | |
| # merge samples |
| for cond, control_idx, scale in zip(controlnet_cond, control_type_idx, conditioning_scale): |
Problem:
The elif isinstance(controlnet_conditioning_scale, list) length check is unreachable after the preceding if isinstance(..., list). In Union variants, some multi-condition paths have no equivalent scale length check.
Impact:
A too-short scale list passes validation, and later zip(...) silently drops later ControlNets or later Union conditions.
Reproduction:
from types import MethodType
import torch
from diffusers import MultiControlNetModel, StableDiffusionControlNetPipeline
class DummyControlNet(torch.nn.Module):
pass
pipe = object.__new__(StableDiffusionControlNetPipeline)
pipe._callback_tensor_inputs = []
pipe.controlnet = MultiControlNetModel([DummyControlNet(), DummyControlNet()])
pipe.check_image = MethodType(lambda self, image, prompt, prompt_embeds: None, pipe)
pipe.check_inputs(
prompt="a prompt",
image=[object(), object()],
callback_steps=None,
callback_on_step_end_tensor_inputs=[],
controlnet_conditioning_scale=[0.5],
control_guidance_start=[0.0, 0.0],
control_guidance_end=[1.0, 1.0],
)
print("No error, but one scale for two ControlNets should be rejected.")Relevant precedent:
Issue #11828 is related to Union scale/list acceptance, but not this silent truncation.
Suggested fix:
if isinstance(controlnet_conditioning_scale, list):
if any(isinstance(i, list) for i in controlnet_conditioning_scale):
raise ValueError("Batched varying conditioning scales are not supported.")
if len(controlnet_conditioning_scale) != len(self.controlnet.nets):
raise ValueError("Scale list length must match the number of ControlNets.")Issue 5: All-zero MultiControlNetUnionModel scales return None
Affected code:
diffusers/src/diffusers/models/controlnets/multicontrolnet_union.py
Lines 52 to 83 in 0f1abc4
| zip(controlnet_cond, control_type, control_type_idx, conditioning_scale, self.nets) | |
| ): | |
| if scale == 0.0: | |
| continue | |
| down_samples, mid_sample = controlnet( | |
| sample=sample, | |
| timestep=timestep, | |
| encoder_hidden_states=encoder_hidden_states, | |
| controlnet_cond=image, | |
| control_type=ctype, | |
| control_type_idx=ctype_idx, | |
| conditioning_scale=scale, | |
| class_labels=class_labels, | |
| timestep_cond=timestep_cond, | |
| attention_mask=attention_mask, | |
| added_cond_kwargs=added_cond_kwargs, | |
| cross_attention_kwargs=cross_attention_kwargs, | |
| from_multi=True, | |
| guess_mode=guess_mode, | |
| return_dict=return_dict, | |
| ) | |
| # merge samples | |
| if down_block_res_samples is None and mid_block_res_sample is None: | |
| down_block_res_samples, mid_block_res_sample = down_samples, mid_sample | |
| else: | |
| down_block_res_samples = [ | |
| samples_prev + samples_curr | |
| for samples_prev, samples_curr in zip(down_block_res_samples, down_samples) | |
| ] | |
| mid_block_res_sample += mid_sample | |
diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_union_sd_xl.py
Lines 1518 to 1522 in 0f1abc4
| if guess_mode and self.do_classifier_free_guidance: | |
| # Inferred ControlNet only for the conditional batch. | |
| # To apply the output of ControlNet to both the unconditional and conditional batches, | |
| # add 0 to the unconditional batch to keep it unchanged. | |
| down_block_res_samples = [torch.cat([torch.zeros_like(d), d]) for d in down_block_res_samples] |
Problem:
MultiControlNetUnionModel.forward() skips every ControlNet whose scale is 0.0. If all scales are zero, it returns (None, None). Union pipelines then iterate over down_block_res_samples in guess-mode CFG.
Impact:
Valid schedules such as control_guidance_start/end outside a step, or explicit controlnet_conditioning_scale=[0.0], can crash in guess mode.
Reproduction:
import torch
from diffusers.models import MultiControlNetUnionModel
class DummyUnion(torch.nn.Module):
config = type("Config", (), {"num_control_type": 6})()
def forward(self, *args, **kwargs):
return [torch.ones(1, 1, 1, 1)], torch.ones(1, 1, 1, 1)
multi = MultiControlNetUnionModel([DummyUnion()])
down, mid = multi(
sample=torch.zeros(1, 4, 8, 8),
timestep=0,
encoder_hidden_states=torch.zeros(1, 1, 4),
controlnet_cond=[torch.zeros(1, 3, 8, 8)],
control_type=[torch.zeros(1, 6)],
control_type_idx=[[0]],
conditioning_scale=[0.0],
return_dict=False,
)
[torch.cat([torch.zeros_like(d), d]) for d in down]Relevant precedent:
MultiControlNetModel does not skip zero scales; it lets the child model return zeroed residual tensors.
Suggested fix:
# Do not skip zero scales. Let the child ControlNet return correctly shaped zero residuals.
# Remove:
if scale == 0.0:
continueIssue 6: ControlNetUnionModel rejects documented bgr channel order
Affected code:
diffusers/src/diffusers/models/controlnets/controlnet_union.py
Lines 143 to 145 in 0f1abc4
| controlnet_conditioning_channel_order (`str`, defaults to `"rgb"`): | |
| The channel order of conditional image. Will convert to `rgb` if it's `bgr`. | |
| conditioning_embedding_out_channels (`tuple[int]`, *optional*, defaults to `(48, 96, 192, 384)`): |
diffusers/src/diffusers/models/controlnets/controlnet_union.py
Lines 608 to 611 in 0f1abc4
| channel_order = self.config.controlnet_conditioning_channel_order | |
| if channel_order != "rgb": | |
| raise ValueError(f"unknown `controlnet_conditioning_channel_order`: {channel_order}") |
Problem:
The config/docstring exposes controlnet_conditioning_channel_order, but Union forward only accepts "rgb" and raises for "bgr".
Impact:
Union behaves inconsistently with ControlNetModel and rejects a documented compatibility mode.
Reproduction:
import torch
from diffusers import ControlNetUnionModel
model = ControlNetUnionModel(
in_channels=4,
conditioning_channels=3,
down_block_types=("DownBlock2D",),
block_out_channels=(8,),
layers_per_block=1,
norm_num_groups=4,
cross_attention_dim=16,
attention_head_dim=1,
addition_time_embed_dim=8,
num_trans_channel=8,
num_trans_head=1,
num_proj_channel=8,
conditioning_embedding_out_channels=(4, 8),
controlnet_conditioning_channel_order="bgr",
)
model(
sample=torch.randn(1, 4, 8, 8),
timestep=0,
encoder_hidden_states=torch.randn(1, 2, 16),
controlnet_cond=[torch.randn(1, 3, 8, 8)],
control_type=torch.zeros(1, 6),
control_type_idx=[0],
return_dict=False,
)Relevant precedent:
diffusers/src/diffusers/models/controlnets/controlnet.py
Lines 654 to 664 in 0f1abc4
| If `return_dict` is `True`, a [`~models.controlnets.controlnet.ControlNetOutput`] is returned, | |
| otherwise a tuple is returned where the first element is the sample tensor. | |
| """ | |
| # check channel order | |
| channel_order = self.config.controlnet_conditioning_channel_order | |
| if channel_order == "rgb": | |
| # in rgb order by default | |
| ... | |
| elif channel_order == "bgr": | |
| controlnet_cond = torch.flip(controlnet_cond, dims=[1]) |
Suggested fix:
if channel_order == "rgb":
pass
elif channel_order == "bgr":
controlnet_cond = [torch.flip(cond, dims=[1]) for cond in controlnet_cond]
else:
raise ValueError(f"unknown `controlnet_conditioning_channel_order`: {channel_order}")Issue 7: Multi-ControlNet wrappers diverge from public API contracts
Affected code:
diffusers/src/diffusers/models/controlnets/multicontrolnet.py
Lines 37 to 73 in 0f1abc4
| controlnet_cond: list[torch.tensor], | |
| conditioning_scale: list[float], | |
| class_labels: torch.Tensor | None = None, | |
| timestep_cond: torch.Tensor | None = None, | |
| attention_mask: torch.Tensor | None = None, | |
| added_cond_kwargs: dict[str, torch.Tensor] | None = None, | |
| cross_attention_kwargs: dict[str, Any] | None = None, | |
| guess_mode: bool = False, | |
| return_dict: bool = True, | |
| ) -> ControlNetOutput | tuple: | |
| for i, (image, scale, controlnet) in enumerate(zip(controlnet_cond, conditioning_scale, self.nets)): | |
| down_samples, mid_sample = controlnet( | |
| sample=sample, | |
| timestep=timestep, | |
| encoder_hidden_states=encoder_hidden_states, | |
| controlnet_cond=image, | |
| conditioning_scale=scale, | |
| class_labels=class_labels, | |
| timestep_cond=timestep_cond, | |
| attention_mask=attention_mask, | |
| added_cond_kwargs=added_cond_kwargs, | |
| cross_attention_kwargs=cross_attention_kwargs, | |
| guess_mode=guess_mode, | |
| return_dict=return_dict, | |
| ) | |
| # merge samples | |
| if i == 0: | |
| down_block_res_samples, mid_block_res_sample = down_samples, mid_sample | |
| else: | |
| down_block_res_samples = [ | |
| samples_prev + samples_curr | |
| for samples_prev, samples_curr in zip(down_block_res_samples, down_samples) | |
| ] | |
| mid_block_res_sample += mid_sample | |
| return down_block_res_samples, mid_block_res_sample |
diffusers/src/diffusers/models/controlnets/multicontrolnet.py
Lines 75 to 176 in 0f1abc4
| def save_pretrained( | |
| self, | |
| save_directory: str | os.PathLike, | |
| is_main_process: bool = True, | |
| save_function: Callable = None, | |
| safe_serialization: bool = True, | |
| variant: str | None = None, | |
| ): | |
| """ | |
| Save a model and its configuration file to a directory, so that it can be re-loaded using the | |
| `[`~models.controlnets.multicontrolnet.MultiControlNetModel.from_pretrained`]` class method. | |
| Arguments: | |
| save_directory (`str` or `os.PathLike`): | |
| Directory to which to save. Will be created if it doesn't exist. | |
| is_main_process (`bool`, *optional*, defaults to `True`): | |
| Whether the process calling this is the main process or not. Useful when in distributed training like | |
| TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on | |
| the main process to avoid race conditions. | |
| save_function (`Callable`): | |
| The function to use to save the state dictionary. Useful on distributed training like TPUs when one | |
| need to replace `torch.save` by another method. Can be configured with the environment variable | |
| `DIFFUSERS_SAVE_MODE`. | |
| safe_serialization (`bool`, *optional*, defaults to `True`): | |
| Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`). | |
| variant (`str`, *optional*): | |
| If specified, weights are saved in the format pytorch_model.<variant>.bin. | |
| """ | |
| for idx, controlnet in enumerate(self.nets): | |
| suffix = "" if idx == 0 else f"_{idx}" | |
| controlnet.save_pretrained( | |
| save_directory + suffix, | |
| is_main_process=is_main_process, | |
| save_function=save_function, | |
| safe_serialization=safe_serialization, | |
| variant=variant, | |
| ) | |
| @classmethod | |
| def from_pretrained(cls, pretrained_model_path: str | os.PathLike | None, **kwargs): | |
| r""" | |
| Instantiate a pretrained MultiControlNet model from multiple pre-trained controlnet models. | |
| The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train | |
| the model, you should first set it back in training mode with `model.train()`. | |
| The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come | |
| pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning | |
| task. | |
| The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those | |
| weights are discarded. | |
| Parameters: | |
| pretrained_model_path (`os.PathLike`): | |
| A path to a *directory* containing model weights saved using | |
| [`~models.controlnets.multicontrolnet.MultiControlNetModel.save_pretrained`], e.g., | |
| `./my_model_directory/controlnet`. | |
| torch_dtype (`torch.dtype`, *optional*): | |
| Override the default `torch.dtype` and load the model under this dtype. | |
| output_loading_info(`bool`, *optional*, defaults to `False`): | |
| Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages. | |
| device_map (`str` or `dict[str, int | str | torch.device]`, *optional*): | |
| A map that specifies where each submodule should go. It doesn't need to be refined to each | |
| parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the | |
| same device. | |
| To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For | |
| more information about each option see [designing a device | |
| map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map). | |
| max_memory (`Dict`, *optional*): | |
| A dictionary device identifier to maximum memory. Will default to the maximum memory available for each | |
| GPU and the available CPU RAM if unset. | |
| low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): | |
| Speed up model loading by not initializing the weights and only loading the pre-trained weights. This | |
| also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the | |
| model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch, | |
| setting this argument to `True` will raise an error. | |
| variant (`str`, *optional*): | |
| If specified load weights from `variant` filename, *e.g.* pytorch_model.<variant>.bin. `variant` is | |
| ignored when using `from_flax`. | |
| use_safetensors (`bool`, *optional*, defaults to `None`): | |
| If set to `None`, the `safetensors` weights will be downloaded if they're available **and** if the | |
| `safetensors` library is installed. If set to `True`, the model will be forcibly loaded from | |
| `safetensors` weights. If set to `False`, loading will *not* use `safetensors`. | |
| """ | |
| idx = 0 | |
| controlnets = [] | |
| # load controlnet and append to list until no controlnet directory exists anymore | |
| # first controlnet has to be saved under `./mydirectory/controlnet` to be compliant with `DiffusionPipeline.from_prertained` | |
| # second, third, ... controlnets have to be saved under `./mydirectory/controlnet_1`, `./mydirectory/controlnet_2`, ... | |
| model_path_to_load = pretrained_model_path | |
| while os.path.isdir(model_path_to_load): | |
| controlnet = ControlNetModel.from_pretrained(model_path_to_load, **kwargs) | |
| controlnets.append(controlnet) | |
| idx += 1 | |
| model_path_to_load = pretrained_model_path + f"_{idx}" | |
| logger.info(f"{len(controlnets)} controlnets loaded from {pretrained_model_path}.") | |
diffusers/src/diffusers/models/controlnets/multicontrolnet_union.py
Lines 47 to 83 in 0f1abc4
| guess_mode: bool = False, | |
| return_dict: bool = True, | |
| ) -> ControlNetOutput | tuple: | |
| down_block_res_samples, mid_block_res_sample = None, None | |
| for i, (image, ctype, ctype_idx, scale, controlnet) in enumerate( | |
| zip(controlnet_cond, control_type, control_type_idx, conditioning_scale, self.nets) | |
| ): | |
| if scale == 0.0: | |
| continue | |
| down_samples, mid_sample = controlnet( | |
| sample=sample, | |
| timestep=timestep, | |
| encoder_hidden_states=encoder_hidden_states, | |
| controlnet_cond=image, | |
| control_type=ctype, | |
| control_type_idx=ctype_idx, | |
| conditioning_scale=scale, | |
| class_labels=class_labels, | |
| timestep_cond=timestep_cond, | |
| attention_mask=attention_mask, | |
| added_cond_kwargs=added_cond_kwargs, | |
| cross_attention_kwargs=cross_attention_kwargs, | |
| from_multi=True, | |
| guess_mode=guess_mode, | |
| return_dict=return_dict, | |
| ) | |
| # merge samples | |
| if down_block_res_samples is None and mid_block_res_sample is None: | |
| down_block_res_samples, mid_block_res_sample = down_samples, mid_sample | |
| else: | |
| down_block_res_samples = [ | |
| samples_prev + samples_curr | |
| for samples_prev, samples_curr in zip(down_block_res_samples, down_samples) | |
| ] | |
| mid_block_res_sample += mid_sample | |
diffusers/src/diffusers/models/controlnets/multicontrolnet_union.py
Lines 86 to 192 in 0f1abc4
| # Copied from diffusers.models.controlnets.multicontrolnet.MultiControlNetModel.save_pretrained with ControlNet->ControlNetUnion | |
| def save_pretrained( | |
| self, | |
| save_directory: str | os.PathLike, | |
| is_main_process: bool = True, | |
| save_function: Callable = None, | |
| safe_serialization: bool = True, | |
| variant: str | None = None, | |
| ): | |
| """ | |
| Save a model and its configuration file to a directory, so that it can be re-loaded using the | |
| `[`~models.controlnets.multicontrolnet.MultiControlNetUnionModel.from_pretrained`]` class method. | |
| Arguments: | |
| save_directory (`str` or `os.PathLike`): | |
| Directory to which to save. Will be created if it doesn't exist. | |
| is_main_process (`bool`, *optional*, defaults to `True`): | |
| Whether the process calling this is the main process or not. Useful when in distributed training like | |
| TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on | |
| the main process to avoid race conditions. | |
| save_function (`Callable`): | |
| The function to use to save the state dictionary. Useful on distributed training like TPUs when one | |
| need to replace `torch.save` by another method. Can be configured with the environment variable | |
| `DIFFUSERS_SAVE_MODE`. | |
| safe_serialization (`bool`, *optional*, defaults to `True`): | |
| Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`). | |
| variant (`str`, *optional*): | |
| If specified, weights are saved in the format pytorch_model.<variant>.bin. | |
| """ | |
| for idx, controlnet in enumerate(self.nets): | |
| suffix = "" if idx == 0 else f"_{idx}" | |
| controlnet.save_pretrained( | |
| save_directory + suffix, | |
| is_main_process=is_main_process, | |
| save_function=save_function, | |
| safe_serialization=safe_serialization, | |
| variant=variant, | |
| ) | |
| @classmethod | |
| # Copied from diffusers.models.controlnets.multicontrolnet.MultiControlNetModel.from_pretrained with ControlNet->ControlNetUnion | |
| def from_pretrained(cls, pretrained_model_path: str | os.PathLike | None, **kwargs): | |
| r""" | |
| Instantiate a pretrained MultiControlNetUnion model from multiple pre-trained controlnet models. | |
| The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train | |
| the model, you should first set it back in training mode with `model.train()`. | |
| The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come | |
| pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning | |
| task. | |
| The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those | |
| weights are discarded. | |
| Parameters: | |
| pretrained_model_path (`os.PathLike`): | |
| A path to a *directory* containing model weights saved using | |
| [`~models.controlnets.multicontrolnet.MultiControlNetUnionModel.save_pretrained`], e.g., | |
| `./my_model_directory/controlnet`. | |
| torch_dtype (`torch.dtype`, *optional*): | |
| Override the default `torch.dtype` and load the model under this dtype. | |
| output_loading_info(`bool`, *optional*, defaults to `False`): | |
| Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages. | |
| device_map (`str` or `dict[str, int | str | torch.device]`, *optional*): | |
| A map that specifies where each submodule should go. It doesn't need to be refined to each | |
| parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the | |
| same device. | |
| To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For | |
| more information about each option see [designing a device | |
| map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map). | |
| max_memory (`Dict`, *optional*): | |
| A dictionary device identifier to maximum memory. Will default to the maximum memory available for each | |
| GPU and the available CPU RAM if unset. | |
| low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): | |
| Speed up model loading by not initializing the weights and only loading the pre-trained weights. This | |
| also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the | |
| model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch, | |
| setting this argument to `True` will raise an error. | |
| variant (`str`, *optional*): | |
| If specified load weights from `variant` filename, *e.g.* pytorch_model.<variant>.bin. `variant` is | |
| ignored when using `from_flax`. | |
| use_safetensors (`bool`, *optional*, defaults to `None`): | |
| If set to `None`, the `safetensors` weights will be downloaded if they're available **and** if the | |
| `safetensors` library is installed. If set to `True`, the model will be forcibly loaded from | |
| `safetensors` weights. If set to `False`, loading will *not* use `safetensors`. | |
| """ | |
| idx = 0 | |
| controlnets = [] | |
| # load controlnet and append to list until no controlnet directory exists anymore | |
| # first controlnet has to be saved under `./mydirectory/controlnet` to be compliant with `DiffusionPipeline.from_prertained` | |
| # second, third, ... controlnets have to be saved under `./mydirectory/controlnet_1`, `./mydirectory/controlnet_2`, ... | |
| model_path_to_load = pretrained_model_path | |
| while os.path.isdir(model_path_to_load): | |
| controlnet = ControlNetUnionModel.from_pretrained(model_path_to_load, **kwargs) | |
| controlnets.append(controlnet) | |
| idx += 1 | |
| model_path_to_load = pretrained_model_path + f"_{idx}" | |
| logger.info(f"{len(controlnets)} controlnets loaded from {pretrained_model_path}.") | |
| if len(controlnets) == 0: | |
| raise ValueError( | |
| f"No ControlNetUnions found under {os.path.dirname(pretrained_model_path)}. Expected at least {pretrained_model_path + '_0'}." |
Problem:
The wrappers accept os.PathLike but concatenate paths with strings. They also accept return_dict=True but always return tuples.
Impact:
Path users get TypeError, and direct model callers do not get the advertised ControlNetOutput.
Reproduction:
from pathlib import Path
from tempfile import TemporaryDirectory
import torch
from diffusers import MultiControlNetModel
from diffusers.models import MultiControlNetUnionModel
class DummyControlNet(torch.nn.Module):
def save_pretrained(self, save_directory, **kwargs):
print(save_directory)
for cls in (MultiControlNetModel, MultiControlNetUnionModel):
with TemporaryDirectory() as tmp:
try:
cls([DummyControlNet()]).save_pretrained(Path(tmp) / "controlnet")
except Exception as e:
print(cls.__name__, type(e).__name__, e)Relevant precedent:
Related closed issue for multi-control save layout: #7814
Suggested fix:
save_directory = os.fspath(save_directory)
...
model_path_to_load = os.fspath(pretrained_model_path)
...
if not return_dict:
return down_block_res_samples, mid_block_res_sample
return ControlNetOutput(
down_block_res_samples=down_block_res_samples,
mid_block_res_sample=mid_block_res_sample,
)Issue 8: Test coverage is missing for several target files
Affected code:
| class StableDiffusionXLControlNetUnionPipeline( |
diffusers/src/diffusers/pipelines/controlnet/pipeline_controlnet_blip_diffusion.py
Line 85 in 0f1abc4
| class BlipDiffusionControlNetPipeline(DeprecatedPipelineMixin, DiffusionPipeline): |
| class FlaxStableDiffusionControlNetPipeline(FlaxDiffusionPipeline): |
Problem:
No tests under tests/pipelines/controlnet, tests/models/controlnets, or tests/single_file reference ControlNetUnionModel, MultiControlNetUnionModel, the Union pipelines, BlipDiffusionControlNetPipeline, FlaxControlNetModel, or FlaxStableDiffusionControlNetPipeline. Slow tests are also missing for SDXL img2img and SDXL inpaint ControlNet files.
Impact:
The confirmed regressions above are not covered by fast tests, and several public or deprecated target pipelines have no slow coverage.
Reproduction:
from pathlib import Path
roots = [Path("tests/pipelines/controlnet"), Path("tests/models/controlnets"), Path("tests/single_file")]
files = [p for root in roots for p in root.rglob("*.py")]
for label, terms in {
"Union": ["ControlNetUnionModel", "MultiControlNetUnionModel", "StableDiffusionXLControlNetUnion"],
"BLIP": ["BlipDiffusionControlNetPipeline"],
"Flax": ["FlaxControlNetModel", "FlaxStableDiffusionControlNetPipeline"],
}.items():
hits = [str(p) for p in files if any(term in p.read_text(encoding="utf-8") for term in terms)]
print(label, hits or "NO TEST REFERENCES")
for p in sorted(Path("tests/pipelines/controlnet").glob("test_controlnet*.py")):
print(p, "@slow" in p.read_text(encoding="utf-8"))Relevant precedent:
PR #10747 introduced MultiControlNetUnionModel; the discussion explicitly called out adding tests, but this checkout has no Union test references.
Suggested fix:
Add fast tests for Union constructor, control_mode=None, multi-union zero scales, scale length validation, top-level import, and save/load PathLike. Add slow Union pipeline tests and slow SDXL img2img/inpaint ControlNet tests. For BLIP-Diffusion ControlNet, either add deprecated-pipeline smoke coverage or document why it is intentionally untested.
Duplicate search status: searched GitHub Issues and PRs for controlnet, ControlNetUnionModel, MultiControlNetUnionModel, control_mode None, addition_time_embed_dim, controlnet_conditioning_scale, bgr, PathLike save_pretrained, BLIP, and Flax. Related items found were #10747, #11828, and #7814, but I did not find exact open duplicates for the issues above.
Activity