Repository navigation
latent_consistency_models model/pipeline review #13637
Description
latent_consistency_models model/pipeline review
Commit tested: 0f1abc4ae8b0eb2a3b40e82a310507281144c423
Review performed against the repository review rules.
Issue 1: Img2Img does not serialize requires_safety_checker
Affected code:
diffusers/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py
Lines 216 to 239 in 0f1abc4
| self.register_modules( | |
| vae=vae, | |
| text_encoder=text_encoder, | |
| tokenizer=tokenizer, | |
| unet=unet, | |
| scheduler=scheduler, | |
| safety_checker=safety_checker, | |
| feature_extractor=feature_extractor, | |
| image_encoder=image_encoder, | |
| ) | |
| if safety_checker is None and requires_safety_checker: | |
| logger.warning( | |
| f"You have disabled the safety checker for {self.__class__} by passing `safety_checker=None`. Ensure" | |
| " that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered" | |
| " results in services or applications open to the public. Both the diffusers team and Hugging Face" | |
| " strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling" | |
| " it only for use-cases that involve analyzing network behavior or auditing its results. For more" | |
| " information, please have a look at https://github.com/huggingface/diffusers/pull/254 ." | |
| ) | |
| self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1) if getattr(self, "vae", None) else 8 | |
| self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor) | |
Problem:
LatentConsistencyModelImg2ImgPipeline.__init__ accepts requires_safety_checker, but never calls self.register_to_config(...). The text2img LCM pipeline does register it, so img2img saved configs lose this user choice.
Impact:
Pipelines constructed with requires_safety_checker=False do not persist that setting through config/save/load paths, causing inconsistent serialization behavior between the two LCM pipelines.
Reproduction:
from diffusers import LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline
kwargs = dict(
vae=None,
text_encoder=None,
tokenizer=None,
unet=None,
scheduler=None,
safety_checker=None,
feature_extractor=None,
requires_safety_checker=False,
)
for cls in (LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline):
pipe = cls(**kwargs)
print(cls.__name__, pipe.config.get("requires_safety_checker"))
# Text2Img prints False; Img2Img prints None.Relevant precedent:
diffusers/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py
Lines 211 to 223 in 0f1abc4
| self.register_modules( | |
| vae=vae, | |
| text_encoder=text_encoder, | |
| tokenizer=tokenizer, | |
| unet=unet, | |
| scheduler=scheduler, | |
| safety_checker=safety_checker, | |
| feature_extractor=feature_extractor, | |
| image_encoder=image_encoder, | |
| ) | |
| self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1) if getattr(self, "vae", None) else 8 | |
| self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor) | |
| self.register_to_config(requires_safety_checker=requires_safety_checker) |
diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py
Lines 314 to 322 in 0f1abc4
| unet=unet, | |
| scheduler=scheduler, | |
| safety_checker=safety_checker, | |
| feature_extractor=feature_extractor, | |
| image_encoder=image_encoder, | |
| ) | |
| self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1) if getattr(self, "vae", None) else 8 | |
| self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor) | |
| self.register_to_config(requires_safety_checker=requires_safety_checker) |
Suggested fix:
self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1) if getattr(self, "vae", None) else 8
self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor)
self.register_to_config(requires_safety_checker=requires_safety_checker)Duplicate search:
No matching issue or PR found for LatentConsistencyModelImg2ImgPipeline requires_safety_checker.
Issue 2: Img2Img accepts a safety checker without a feature extractor
Affected code:
diffusers/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py
Lines 216 to 239 in 0f1abc4
| self.register_modules( | |
| vae=vae, | |
| text_encoder=text_encoder, | |
| tokenizer=tokenizer, | |
| unet=unet, | |
| scheduler=scheduler, | |
| safety_checker=safety_checker, | |
| feature_extractor=feature_extractor, | |
| image_encoder=image_encoder, | |
| ) | |
| if safety_checker is None and requires_safety_checker: | |
| logger.warning( | |
| f"You have disabled the safety checker for {self.__class__} by passing `safety_checker=None`. Ensure" | |
| " that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered" | |
| " results in services or applications open to the public. Both the diffusers team and Hugging Face" | |
| " strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling" | |
| " it only for use-cases that involve analyzing network behavior or auditing its results. For more" | |
| " information, please have a look at https://github.com/huggingface/diffusers/pull/254 ." | |
| ) | |
| self.vae_scale_factor = 2 ** (len(self.vae.config.block_out_channels) - 1) if getattr(self, "vae", None) else 8 | |
| self.image_processor = VaeImageProcessor(vae_scale_factor=self.vae_scale_factor) | |
diffusers/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py
Lines 494 to 506 in 0f1abc4
| # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.run_safety_checker | |
| def run_safety_checker(self, image, device, dtype): | |
| if self.safety_checker is None: | |
| has_nsfw_concept = None | |
| else: | |
| if torch.is_tensor(image): | |
| feature_extractor_input = self.image_processor.postprocess(image, output_type="pil") | |
| else: | |
| feature_extractor_input = self.image_processor.numpy_to_pil(image) | |
| safety_checker_input = self.feature_extractor(feature_extractor_input, return_tensors="pt").to(device) | |
| image, has_nsfw_concept = self.safety_checker( | |
| images=image, clip_input=safety_checker_input.pixel_values.to(dtype) | |
| ) |
Problem:
The img2img constructor does not reject safety_checker != None with feature_extractor=None. Later, run_safety_checker unconditionally calls self.feature_extractor(...), producing a late TypeError.
Impact:
Users can construct an invalid pipeline successfully and only fail during inference/safety checking with an unclear 'NoneType' object is not callable error.
Reproduction:
import torch
from diffusers import LatentConsistencyModelImg2ImgPipeline
class DummySafetyChecker:
def __call__(self, images, clip_input):
return images, [False] * images.shape[0]
pipe = LatentConsistencyModelImg2ImgPipeline(
vae=None,
text_encoder=None,
tokenizer=None,
unet=None,
scheduler=None,
safety_checker=DummySafetyChecker(),
feature_extractor=None,
requires_safety_checker=True,
)
try:
pipe.run_safety_checker(torch.zeros(1, 3, 8, 8), "cpu", torch.float32)
except Exception as e:
print(type(e).__name__, str(e))
# TypeError 'NoneType' object is not callableRelevant precedent:
diffusers/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py
Lines 205 to 209 in 0f1abc4
| if safety_checker is not None and feature_extractor is None: | |
| raise ValueError( | |
| "Make sure to define a feature extractor when loading {self.__class__} if you want to use the safety" | |
| " checker. If you do not want to use the safety checker, you can pass `'safety_checker=None'` instead." | |
| ) |
diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py
Lines 279 to 283 in 0f1abc4
| if safety_checker is not None and feature_extractor is None: | |
| raise ValueError( | |
| "Make sure to define a feature extractor when loading {self.__class__} if you want to use the safety" | |
| " checker. If you do not want to use the safety checker, you can pass `'safety_checker=None'` instead." | |
| ) |
Suggested fix:
if safety_checker is not None and feature_extractor is None:
raise ValueError(
"Make sure to define a feature extractor when loading {self.__class__} if you want to use the safety"
" checker. If you do not want to use the safety checker, you can pass `'safety_checker=None'` instead."
)Duplicate search:
No matching issue or PR found for LatentConsistencyModelImg2ImgPipeline feature_extractor.
Issue 3: LCM pipeline __call__ docs have stale parameters and defaults
Affected code:
diffusers/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py
Lines 642 to 690 in 0f1abc4
| def __call__( | |
| self, | |
| prompt: str | list[str] = None, | |
| height: int | None = None, | |
| width: int | None = None, | |
| num_inference_steps: int = 4, | |
| original_inference_steps: int = None, | |
| timesteps: list[int] = None, | |
| guidance_scale: float = 8.5, | |
| num_images_per_prompt: int | None = 1, | |
| generator: torch.Generator | list[torch.Generator] | None = None, | |
| latents: torch.Tensor | None = None, | |
| prompt_embeds: torch.Tensor | None = None, | |
| ip_adapter_image: PipelineImageInput | None = None, | |
| ip_adapter_image_embeds: list[torch.Tensor] | None = None, | |
| output_type: str | None = "pil", | |
| return_dict: bool = True, | |
| cross_attention_kwargs: dict[str, Any] | None = None, | |
| clip_skip: int | None = None, | |
| callback_on_step_end: Callable[[int, int], None] | None = None, | |
| callback_on_step_end_tensor_inputs: list[str] = ["latents"], | |
| **kwargs, | |
| ): | |
| r""" | |
| The call function to the pipeline for generation. | |
| Args: | |
| prompt (`str` or `list[str]`, *optional*): | |
| The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`. | |
| height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): | |
| The height in pixels of the generated image. | |
| width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): | |
| The width in pixels of the generated image. | |
| num_inference_steps (`int`, *optional*, defaults to 50): | |
| The number of denoising steps. More denoising steps usually lead to a higher quality image at the | |
| expense of slower inference. | |
| original_inference_steps (`int`, *optional*): | |
| The original number of inference steps use to generate a linearly-spaced timestep schedule, from which | |
| we will draw `num_inference_steps` evenly spaced timesteps from as our final timestep schedule, | |
| following the Skipping-Step method in the paper (see Section 4.3). If not set this will default to the | |
| scheduler's `original_inference_steps` attribute. | |
| timesteps (`list[int]`, *optional*): | |
| Custom timesteps to use for the denoising process. If not defined, equal spaced `num_inference_steps` | |
| timesteps on the original LCM training/distillation timestep schedule are used. Must be in descending | |
| order. | |
| guidance_scale (`float`, *optional*, defaults to 7.5): | |
| A higher guidance scale value encourages the model to generate images closely linked to the text | |
| `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`. | |
| Note that the original latent consistency models paper uses a different CFG formulation where the |
diffusers/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_img2img.py
Lines 711 to 760 in 0f1abc4
| def __call__( | |
| self, | |
| prompt: str | list[str] = None, | |
| image: PipelineImageInput = None, | |
| num_inference_steps: int = 4, | |
| strength: float = 0.8, | |
| original_inference_steps: int = None, | |
| timesteps: list[int] = None, | |
| guidance_scale: float = 8.5, | |
| num_images_per_prompt: int | None = 1, | |
| generator: torch.Generator | list[torch.Generator] | None = None, | |
| latents: torch.Tensor | None = None, | |
| prompt_embeds: torch.Tensor | None = None, | |
| ip_adapter_image: PipelineImageInput | None = None, | |
| ip_adapter_image_embeds: list[torch.Tensor] | None = None, | |
| output_type: str | None = "pil", | |
| return_dict: bool = True, | |
| cross_attention_kwargs: dict[str, Any] | None = None, | |
| clip_skip: int | None = None, | |
| callback_on_step_end: Callable[[int, int], None] | None = None, | |
| callback_on_step_end_tensor_inputs: list[str] = ["latents"], | |
| **kwargs, | |
| ): | |
| r""" | |
| The call function to the pipeline for generation. | |
| Args: | |
| prompt (`str` or `list[str]`, *optional*): | |
| The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`. | |
| height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): | |
| The height in pixels of the generated image. | |
| width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`): | |
| The width in pixels of the generated image. | |
| num_inference_steps (`int`, *optional*, defaults to 50): | |
| The number of denoising steps. More denoising steps usually lead to a higher quality image at the | |
| expense of slower inference. | |
| original_inference_steps (`int`, *optional*): | |
| The original number of inference steps use to generate a linearly-spaced timestep schedule, from which | |
| we will draw `num_inference_steps` evenly spaced timesteps from as our final timestep schedule, | |
| following the Skipping-Step method in the paper (see Section 4.3). If not set this will default to the | |
| scheduler's `original_inference_steps` attribute. | |
| timesteps (`list[int]`, *optional*): | |
| Custom timesteps to use for the denoising process. If not defined, equal spaced `num_inference_steps` | |
| timesteps on the original LCM training/distillation timestep schedule are used. Must be in descending | |
| order. | |
| guidance_scale (`float`, *optional*, defaults to 7.5): | |
| A higher guidance scale value encourages the model to generate images closely linked to the text | |
| `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`. | |
| Note that the original latent consistency models paper uses a different CFG formulation where the | |
| guidance scales are decreased by 1 (so in the paper formulation CFG is enabled when `guidance_scale > |
Problem:
Both __call__ docstrings say num_inference_steps defaults to 50 and guidance_scale defaults to 7.5, while the signatures default to 4 and 8.5. The img2img docstring also documents nonexistent height and width parameters and omits its real image and strength parameters.
Impact:
The generated API docs mislead users about LCM's fast-step defaults and img2img inputs.
Reproduction:
import inspect
from diffusers import LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline
for cls in (LatentConsistencyModelPipeline, LatentConsistencyModelImg2ImgPipeline):
sig = inspect.signature(cls.__call__)
doc = inspect.getdoc(cls.__call__) or ""
print(cls.__name__, sig.parameters["num_inference_steps"].default, sig.parameters["guidance_scale"].default)
print("doc says steps default 50:", "defaults to 50" in doc)
print("doc says guidance default 7.5:", "defaults to 7.5" in doc)
img_doc = inspect.getdoc(LatentConsistencyModelImg2ImgPipeline.__call__) or ""
print("img2img signature has image:", "image" in inspect.signature(LatentConsistencyModelImg2ImgPipeline.__call__).parameters)
print("img2img docs image:", "image (`" in img_doc)
print("img2img docs height:", "height (`int`" in img_doc)Relevant precedent:
diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py
Lines 891 to 902 in 0f1abc4
| image (`torch.Tensor`, `PIL.Image.Image`, `np.ndarray`, `list[torch.Tensor]`, `list[PIL.Image.Image]`, or `list[np.ndarray]`): | |
| `Image`, numpy array or tensor representing an image batch to be used as the starting point. For both | |
| numpy array and pytorch tensor, the expected value range is between `[0, 1]` If it's a tensor or a list | |
| or tensors, the expected shape should be `(B, C, H, W)` or `(C, H, W)`. If it is a numpy array or a | |
| list of arrays, the expected shape should be `(B, H, W, C)` or `(H, W, C)` It can also accept image | |
| latents as `image`, but if passing latents directly it is not encoded again. | |
| strength (`float`, *optional*, defaults to 0.8): | |
| Indicates extent to transform the reference `image`. Must be between 0 and 1. `image` is used as a | |
| starting point and more noise is added the higher the `strength`. The number of denoising steps depends | |
| on the amount of noise initially added. When `strength` is 1, added noise is maximum and the denoising | |
| process runs for the full number of iterations specified in `num_inference_steps`. A value of 1 | |
| essentially ignores `image`. |
Suggested fix:
Update the LCM docstrings to match their signatures: num_inference_steps default 4, guidance_scale default 8.5, and for img2img replace the stale height/width entries with image and strength documentation.
Duplicate search:
No matching issue or PR found for LCM img2img docstring/default mismatches.
Coverage and duplicate-search status:
Fast and slow tests exist for both LCM pipelines under tests/pipelines/latent_consistency_models/; slow coverage is not missing. Local target pytest collection was blocked by the venv torch build missing torch._C._distributed_c10d via shared test utilities. python utils/check_copies.py passed. Broad duplicate searches found historical LCM issues/PRs, but no duplicates for the three findings above.
Activity
Lucky-Forrest923 commentedon Oct 9, 2026
Hi @hlky — I'd like to work on this one, if it's open for community contribution.
I reproduced Issue 1 locally:
LatentConsistencyModelImg2ImgPipelinereturnsNoneforrequires_safety_checkerwhere the text2img pipeline returnsFalse. I also confirmed thenum_inference_steps/guidance_scaledefaults mismatch from Issue 3 on both LCM pipelines.I plan to fix all findings in one PR. Let me know if you'd rather I split these up — otherwise I'll do them in one PR.
hlky commentedon Oct 9, 2026
Thanks @Lucky-Forrest923 for reproducing these and offering to work on them. As the issue reporter, one focused PR covering these related findings seems reasonable to me; I don’t have a preference for splitting them. I’ll leave the preferred scope and acceptance decisions to upstream maintainers.