StableDiffusionPipeline produces nans for SD2.1 (Mac, cpu device)

@yiyixuxu

Describe the bug

StableDiffusionPipeline produces nans for SD2.1.

The issue seems to be in the pipe.unet.conv_in. Printing the mean / std for sample before and after conv_in gives:

m,s = tensor(0.005) tensor(0.996)
m,s = tensor(15962061100191580160., grad_fn=<MeanBackward0>) tensor(16510648845702789070848., grad_fn=<StdBackward0>)

Reproduction

Make sure I'm using the latest official diffusers version

# uninstall local, editable version
!pip uninstall diffusers -y -qq
# install official version
!pip install diffusers -qq
# restart notebook now

Code to reproduce

from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained('stabilityai/stable-diffusion-2-1')
result = pipe('the sun', output_type='pt')
result.images[0]

Output:

tensor([[[nan, nan, nan,  ..., nan, nan, nan],
         [nan, nan, nan,  ..., nan, nan, nan],
         [nan, nan, nan,  ..., nan, nan, nan],
...

Logs

No response

System Info

diffusers version: 0.24.0
Platform: macOS-14.1.2-arm64-arm-64bit
Python version: 3.11.6
PyTorch version (GPU?): 2.1.1 (False)
Huggingface_hub version: 0.19.4
Transformers version: 4.35.2
Accelerate version: 0.25.0
xFormers version: not installed
Using GPU in script?: no
Using distributed or parallel set-up in script?: no

Who can help?

Questions on pipelines > Stable Diffusion: @yiyixuxu @DN6 @sayakpaul @patrickvonplaten

@UmerHA

Hi @UmerHA I was able to reproduce this. Using the MPS backend seems to resolve the issue pipe.to("mps"). Although I will look into why this is happening on CPU.

@pcuenca

cc: @pcuenca

@UmerHA

Hi @UmerHA you are right, the pipeline is unstable using CPU. I tried to isolate a minimal reproduction snippet but couldn't. In the meantime, is there any reason why you are using the cpu instead of the mps device?

@pcuenca

Hi @pcuenca, for educational purposes only. I created a very slim version of diffusers without all optimizations and wanted to compare results with the original diffusers.

Understood. What I saw is that calling self.conv_in(sample) produced different results after moving sample and conv_in to mps. However, attempting to reproduce the same case in an isolated script worked fine, using the same convolution weights (I saved them to a numpy array).