Repository navigation
StableDiffusionPipeline producing unexpected output with MPS device using diffusers==0.4.0 #760
Description
Describe the bug
I tried testing the potential speed updates of diffusers 0.4.0 on my M1 mac using an existing StableDiffusionPipeline-based script, and I found that a large image that would take ~3 min to generate in diffusers 0.3.0 was estimated to take more than 10x as long.
Since my existing script had a lot going on (e.g. large resolutions, attention slicing), I tried to diagnose the problem with a minimal script (see below), running in two identical environments, with the only difference being the diffusers version.
In diffusers 0.3.0, it takes ~35 seconds to generate a reasonable result like this:
In diffusers 0.4.0, it takes ~50 seconds (which is slower than 0.3.0, but better that the 10x performance hit I was getting before), but each attempt (with varying seeds) triggered the NSFW filter. Disabling the filter, the results appear to be just noise:
I'm not sure if the initial 10x performance hit I initially observed in my original script would be fixed by fixing this bug, but this certainly seems to be at least part of it.
Reproduction
import torch
from diffusers import StableDiffusionPipeline
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
pipe.to("mps")
result = pipe("dogs playing poker", generator=torch.manual_seed(1))
result.images[0].save("test.png")
Logs
Under 0.4.0 there's also this warning:
/opt/homebrew/Caskroom/miniforge/base/envs/sd/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py:222: UserWarning: The operator 'aten::repeat_interleave.self_int' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
text_embeddings = text_embeddings.repeat_interleave(num_images_per_prompt, dim=0)System Info
diffusersversion: 0.4.0- Platform: macOS-12.6-arm64-arm-64bit
- Python version: 3.10.6
- PyTorch version (GPU?): 1.13.0.dev20220911 (False)
- Huggingface_hub version: 0.10.0
- Transformers version: 4.21.3
- Using GPU in script?: MPS
- Using distributed or parallel set-up in script?: no
Activity
pcuenca commentedon Oct 7, 2022
Hi @jncasey, thanks for reporting this!
From the details you describe it looks like one of the last-minute optimizations we did for the release is not compatible with
mps, so it's falling back to the CPU and that's why it's slower. It's probably the reason why it fails to generate images, I'll test and work on a fix today.I do apologize for the trouble, I tested on my Mac a few hours before the release, but not after all the changes were merged together. Thanks again for letting us know!
pcuenca commentedon Oct 7, 2022
(For reference, I believe this is what's affecting
mps: #741 (comment))pcuenca commentedon Oct 7, 2022
Hi @jncasey! #766 will replace the slow operation (once it's approved and released), but I had to upgrade my PyTorch nightly version to the latest one (
1.14.0.dev20221007) to fix the broken images issue. Would it be possible for you to do the same, following the instructions here?Thank you!
jncasey commentedon Oct 7, 2022
Okay, so updating to the latest daily pytorch has solved the noise issue – the image generated by my test script with 0.4.0 is now reasonable and very similar (thought not identical) to the one generated by the old pytorch and 0.3.0 using the same seed. The speed under 0.4.0 remains roughly the same.
Interestingly, with this pytorch, my original script now seems to be slower under 0.3.0 than even it was under 0.4.0
0.3.0, pytorch 1.13.0.dev20220911: ~3 min
0.4.0, pytorch 1.13.0.dev20220911: ~30 min
0.3.0, pytorch 1.14.0.dev20221007: ~40 min
0.4.0, pytorch 1.14.0.dev20221007: ~24 min
I have to run to work now, so I can't dig any deeper, but I'll keep poking around this weekend.
tcapelle commentedon Oct 7, 2022
I was expecting faster performance with the new optimization, but the exported CoreML version of the Unet (after tracing) is roughly the same on my machine.
More info here http://wandb.me/mac_diffusion
pcuenca commentedon Oct 7, 2022
HI @jncasey, the change in #766 has not been released yet, it may help with performance as the operation should not be moved to the CPU. However, it looks like the version of PyTorch has had a big impact. I haven't observed that in my system, so I'm not sure what could be going on.
pcuenca commentedon Oct 7, 2022
Hi @tcapelle! Your analyses on this matter are always very interesting, I have yet to try your most recent one on my system. I'm slightly surprised about your findings of
mps-PyTorch vs Tensorflow, even considering that PyTorch is a pre-release. I suspect that memory might be a factor, but we should find out for sure as my computer has 64 GB.Regarding the problems mentioned in this issue, I'd recommend you use the branch in #766, which is still not part of
v0.4.1. I'm not sure what you mean by your Core ML metrics with respect todiffusers, as we are usingmpsfor now. I'll review your findings tomorrow and try to reproduce them :)jncasey commentedon Oct 7, 2022
Thanks, @pcuenca. That certainly seems to be the case. I'll keep poking around this weekend and report back if I can figure anything out. Thankfully, I've got access to another machine with a 3090, so I can work around the mps issues.
jncasey commentedon Oct 8, 2022
Okay, here are a few more data points. I haven't yet tried #766 yet, but I've done some methodical (if not totally scientific) tests using the simple script in the first post, comparing
diffusers 0.3.0and0.4.1, and using my originaltorch 1.13.0.dev20220911, the latest1.14.0.dev20221007, and1.13.0.dev20221006(to compare 1.13 and 1.14 nightlies from similar dates). These were all run in the same environment, and using pip to change the versions up and down. The times were repeatable no matter which order the versions were upgraded or downgraded.diffusers 0.3.0,torch 1.13.0.dev2022911averaged 0:33 over five runs, and produced the image in the first post.diffusers 0.4.1,torch 1.13.0.dev2022911averaged 0:47 over five runs, and produced the noise in the first post.There was no difference in time between the two more recent torch nightlies when using the same version of diffusers.
diffusers 0.3.0, and either of the recent torch nightlies averaged 3:58 over five runs.diffusers 0.4.1, and either of the recent torch nightlies averaged 0:45 over five runs.All four of those runs produced identical images:
Which is slightly different from the image generated with
diffusers 0.3.0,torch 1.13.0.dev2022911:I don't know what any of this means, but it's clear that something very different was happening with
diffusers 0.3.0andtorch 1.13.0.dev2022911that resulted in different math but much faster execution.patrickvonplaten commentedon Oct 10, 2022
Thanks for the very in-detail benchmarking here @jncasey !
Seems like things are still a bit unstable both from the
diffusersand native PyTorch side. I guess we have to opt always for the newest PyTorch nightly.jncasey commentedon Oct 13, 2022
Thanks, @pcuenca! FYI it looks like the other Stable Diffusion pipelines (e.g. Img2Img, inpaint) are still using
repeat_interleave, so they may want your fix, too.The change hasn't returned performance to what I'm getting with diffusers 0.3.0 and the torch nightly from September 11, but that seems to be on the pytorch side based on this other report.
FahimF commentedon Oct 14, 2022
@jncasey Actually, the performance on the diffusers 0.4.0 branch became horrible way before the issues on the PyTorch side ... I filed a bug report about the 0.4.0 branch issues about a month ago:
#548
The issues on the PyTorch side started only around the end of September - I believe the 25th to be exact ... So this might be two separate issues. But have not heard anything about my report about 0.4.0 performance since I opened it, I'm afraid ... 😄
So personally, I'm staying away from anything after diffusers 0.3.0 on the diffusers side and anything after 1.13.0.dev20220924 on the PyTorch side since performance is terrible with any combination of the two after those builds on MPS.
patrickvonplaten commentedon Oct 14, 2022
Hey @FahimF,
Could you quantify "horrible" a bit with benchmark numbers?
Note that MPS is still not stable yet on PyTorch. We're in contact with the Pytorch team to make things more stable, but note that for now we will always make our code be compatible with the newest PyTorch versions. Also cc @pcuenca
FahimF commentedon Oct 14, 2022
@patrickvonplaten Not blaming you guys, just noting what was previously reported 😄 Please look at the bug report I filed about a month back (#548) for numbers, which I already provided.
The numbers might have changed since the 0.4.0 (and subsequent releases) but I have not done actual numbers since then. I do install the latest version every once in a while (on top of PyTorch 1.13.0.dev20220924 since after that PyTorch slows down a lot too — the numbers for that are here: pytorch/pytorch#86048) and test when I can but it continues to be slow.
Do note, I do appreciate everything you guys are doing and am simply reporting issues as I find them since I do realize that not everybody uses MPS. I wish I could do more by helping with the code, but I'm not up to the level of working on the code at the diffusers level - doing a GUI is about all I can do at this point 😄
But if there's anything I can do to help (since I do work with MPS day in and day out and do work on Stable Diffusion related stuff whenever I can) do let me know.
patrickvonplaten commentedon Oct 17, 2022
Sorry, I didn't mean to sound offended here 😅 By actively filing issues, you're helping the library a lot - please continue to do so!
I was just wondering if you had numbers with versions that we could easily reproduce. Generally, both
diffusersand PyTorch's MPS are changing so fast that it's hard to have reliable benchmarks, but it's great to point out if something has slowed down - thanks a lot for this!FahimF commentedon Oct 17, 2022
Apologies, didn't mean to indicate that you were offended — in fact, I was explaining to indicate that I wasn't offended/blaming you guys 😄
Regarding numbers and versions, I thought I did provide those but I do tend to be a bit verbose and so the way I provide the info might not be very clear. Please let me know what would make things clearer, or if perhaps I can provide more info and I'd be happy to do so.
I did provide the latest numbers to @pcuenca with versions on #548 but will reproduce here in more concise form in case it's helpful. Please let me know if you'd prefer something like this or if I'm missing other useful info and I'd be happy to elaborate.
These tests were done yesterday to compare diffusers 0.3.0 and 0.5.1 and both used PyTorch 1.13.0.dev20220924 so as to eliminate the slowdown issues that come from the PyTorch side (PyTorch became slower on MPS only after 25-09-2022). Each test was for 50 steps generating the same prompt.
diffusers 0.5.1
512 x 512 = 57s
768 x 512 = 19 mins 17s
diffusers 0.3.0
512 x 512 = 32s
768 x 512 = 58s
Do note that with the early 0.4.0 builds (before it was released) a 512 x 512 image used to take about 80s but it seemed faster with 0.5.1 when I tested yesterday. The numbers are based off of a single run and are not averaged over several runs.
Also, I did monitor the CPU/GPU while the above runs were going on and it looks as if 0.5.1 uses both the CPU and the GPU equally with more emphasis on the CPU while 0.3.0 uses the GPU lot more (especially for the 768 x 512 image) and finishes extremely quickly.
Add Resnet50 fp16 variant to pytests. (huggingface#760)