Apple MPS image generation times take much longer with 0.4.0

Describe the bug

It used to take about 90 seconds to generate a 768 x 512 image with diffusers 0.3.0. I checked out the main branch with 0.4.0dev0 today and the time has increased to about 600 seconds and that too for only 20 steps, as opposed to 75 steps initially. It feels as if this build uses both the CPU and GPU and relies more on the CPU but is mostly based on gut feel, the fact that the system UI is less responsive and the fact that the Activity Monitor only shows about 50% GPU usage at most times ...

So has something changed? Performance under Apple Silicon has gotten worse though it is much more stable. Do note that 512 x 512 image generation still seems to work at about the same speed but if you try 768 x 512 or 768 x 768, the image generation slows down a lot.

Update: I was wrong about 512 x 512 not being affected. That too has slowed down but not as much. A 512 x 512 image used to take around 55 second with 0.3.0 but now it takes around 80 seconds.

Reproduction

Just run an image generation with Stable Diffusion but for an image size other than 512 x 512. I've tested for both 768 x 768 and 768 x 512 and the results were really slow.

Logs

No response

System Info

diffusers 0.4.0dev0
Apple M1 Max
Monterey 12.6

@pcuenca

Gently pinging @pcuenca here :-)

(sorry for the long wait @FahimF)

@pcuenca maybe we should post a longer "what's the current state" message in #760 ?

@FahimF

Hi @FahimF my apologies for being so slow! I need to seriously test the latest pre-release versions of PyTorch and try to find what the problem might be. I hope to provide a better update on Monday. Thanks for your patience, and also for your help in reporting these problems, they are very helpful even if we are slow sometimes.

@pcuenca

@pcuenca Not blaming you guys, at all! Much appreciate what you are doing since I did try working with the Tensorflow version of Stable Diffusion (because it is much faster than PyTorch and Hugging Face Diffusers at the moment) and I must confess that I much prefer working with PyTorch and Hugging Face Diffusers if I can 😄

So, really appreciate what you guys are doing and am simply reporting issues as I see them since I do work with PyTorch MPS stuff almost every day. Just not at the coding level I'd like, to be able to help out. But am trying to learn more so that I can help out .. especially since PyTorch has some open tickets for adding missing MPS functionality which are labeled as "Good first issues". But not there yet since I get distracted by working with Stable Diffusion 😄

Anyway, if you want numbers, tests, or further information, please let me know. I'd be happy to provide more info. I also just noticed that you guys are up to 0.5.1 now. I don't know if I tested the latest releases - I do know that I tested 0.4.0 after it was released and it was still slow. Will test and update this ticket if the situation has changed.

@pcuenca

@pcuenca Tried testing with 0.5.1 to see if there's a change with the speed issues but can't even get simple code to run using 0.5.1 with diffusers due to a persistent crash. Filed a separate bug report about that here: #852

@pcuenca

@pcuenca Tested with the latest diffusers (0.5.1) and the pre-slowdown version of PyTorch nightly (1.13.0.dev20220924) and it appears that diffusers is now almost as fast as it used to be. So there's some improvement over the initial 0.4.0 pre-release builds I last tested on but slightly slower than it used to be ...

But for anything other than 512 x 512 images, it's still extremely slow and that is not the behaviour you could see with diffusers 0.3.0 ... I tested generating a 768 x 512 image and that took almost 20 minutes (19 mins 17 sec) and is far worse than even my last run ☹️

To compare, I reverted back to diffusers 0.3.0 and ran an image generation for a 768 x 512 image and it completed in 58 seconds!

I monitored CPU/GPU usage for both of the above image generations and for the diffusers 0.5.1 run, the GPU usage never went above 60% while the CPU usage climbed to over 90% - 100% at times. But for diffusers 0.3.0, the CPU usage remained relatively low while the GPU usage was always around 98% during the image generation. So it appears that since 0.3.0, some tasks have switched to CPU instead of GPU and this is why things are slower?

@FahimF

Hi @FahimF that appears to be the case, absolutely!

I have of couple of not-yet-updates:

Some of the workarounds we added for unsupported ops are no longer necessary with the combination of PyTorch 1.13 pre-RC and the latest macOS Ventura beta. I think most were solved by the PyTorch team, while Apple fixed part of the array size limitations that caused weird crashes and inconsistencies (some of them are still failing though). I did not notice much of a performance improvement in my tests, but I didn't test non-standard sizes. I'll do it tomorrow, and I'll post a branch in case you'd like to test too.
The PyTorch team is hard at work trying to fix some performance degradation issues that affect MPS, see for example: [einsum] Call view instead of sum to remediate MPS regression pytorch/pytorch#87135. I'm not sure these fixes will make it into the final release, but it's looking great :)
Another factor I need to look into is the impact of system memory size on these problems. My test computer has 64 GB of RAM, which is a lot. I've always suspected that a system with less RAM might reach a cliff of heavy degradation if it needs to page/swap. The kind of performance you describe (fast for 512, extremely slow for larger sizes) could be indicative of that sort of problem, but it could also be explained by delegating work to the CPU. Maybe both are related. I'll do some testing here.

I hope these directions convert into some real updates soon! Thanks a lot for your patience!

@pcuenca

Hey @pcuenca, thanks for taking a look and for the detailed notes — much appreciated 😄

Didn't even know that there was a 1.13 pre-RC for PyTorch. I did notice that the nightly dev versions changed to 1.14 and that there was no 1.13 released but didn't realize what it signified. Will hunt down the RC and test with it. I haven't switched to Ventura yet, but since it will release next week, it shouldn't be long ...
Thank you for the link to performance fixes on the PyTorch side. I've been looking for any indications of performance fixes and was following several threads but hadn't seen that one. Looks interesting!
Regarding memory size, two things — 1. I have 32GB of RAM on my machine 2. A larger size image (768 x 512) completes in 58 seconds with diffusers 0.3.0 but takes almost 20 minutes with 0.5.1. So would memory be the cause there? Would it not seem unlikely since it works fine with the older version of diffusers?

Thank you again for your update, the testing, and all that you do, and if you need me to test anything at any point or provide further info, please do let me know.

@FahimF

Hey @FahimF, small update. I tested diffusers 0.3.0 and this branch: #926, both on PyTorch "pre RC", which you can install like so:

pip install --pre torch --extra-index-url https://download.pytorch.org/whl/test/cpu

(It uses test instead of nightly in the URL).

These are my results:

Width	Diffusers	Time
768	0.3.0	101.47
	m1-torch-1.13	83.02
512	0.3.0	44.80
	m1-torch-1.13	39.95

I'm using a small warmup of 10 steps in both cases. This is the code I'm testing with:

from diffusers import StableDiffusionPipeline
import time
import torch

device = "mps"
seed = 1337

sdm = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4",
)
sdm = sdm.to(device)
    
prompt = "A painting of a squirrel eating a burger"

# Warmup
torch.manual_seed(seed)
images = sdm(prompt, num_inference_steps=10).images

# Actual inference
torch.manual_seed(seed)
start = time.time()
images = sdm(prompt).images
print(f"[{device}] - {time.time()-start:.2f}")

It'd be really interesting to know if you still see that dramatic difference you described when running at a size larger than 512. Please, let me know if you have any questions or insights :) Thanks a lot!

@pcuenca

Hey @pcuenca, will do the testing either tonight or tomorrow morning using your script for all variations - without PyTorch 13 pre-RC, with it, and for the two different image sizes and let you know what the figures are 😄

Awesome, thanks a lot! 😄

@pcuenca

@pcuenca here are the results:

macOS version is 12.6

RAM on device: 32GB

The height is 512 for all the following tests and the Python version is 3.9.13 and the warm up run always did 512 x 512 images but for 10 steps.

PyTorch versions used are 1.13.0.dev20220924 (before PyTorch image generation slowed down) and PyTorch 1.13.0 (pre RC).

Installed 1.13.0.dev20220924 using: pip install --upgrade --pre torch==1.13.0.dev20220924 --extra-index-url https://download.pytorch.org/whl/nightly/cpu

Installed 1.13.0 using: pip install --pre torch --extra-index-url https://download.pytorch.org/whl/test/cpu

Configuration	width = 512	width = 768
torch 1.13.0.dev20220924 diffusers 0.3.0	Warm up - 10s Full - 32.02s	Warm up - 13s 56.57s
torch 1.13.0.dev20220924 diffusers 0.6.0	Warm up - 13s Full - 41s	Warm up - 12s Full - 36m 45s
torch 1.13.0 diffusers 0.3.0	Warm up - 1m 40s Full - 6m 57s	Warm up - 1m 37s Full - 30m 16s
torch 1.13.0 diffusers 0.6.0	Warm up - 14s Full - 58s	Warm up - 14s Full - 7m 37s

As you'll note, the best times were for the older PyTorch 1.13.0 dev release and diffusers 0.3.0. The latest diffusers and the PyTorch pre RC does seem to improve timing but there's a weird outlier in terms of generation time for diffusers 0.3.0 and the 1.13.0 pre RC for 512 images. I ran most of these last night before went to bed. If you need me to re-run any of them, please let me know ...

@FahimF

Thanks a lot @FahimF, really appreciated! I'll try to get a M1 machine with macOS 12.6 to see what impact the OS has.

@pcuenca

You're welcome, @pcuenca 🙂 If it would help, I can upgrade to Ventura and test again. Let me know. But since I've only got the one machine (or at least, the one machine with an M1 chip), I'd like to be in a position where I'd be most useful for you for testing. So let me know what works better. I mean if further testing/timing would be useful ...

@FahimF

Hi @FahimF, I got a Mac mini M1 with 16 GB running macOS 12.6 from a cloud service, and it certainly looks like memory has something to do with it! The PyTorch and diffusers versions are important too, and macOS itself might be a factor too. These are some of my measures on this machine (only for the warmup phase):

		Warmup	+attention_slicing
PyTorch 1.13.0	0.3.0	Much slower than ->	4:26
	main	4:34	1:13
	m1-torch-1.13	3:08	0:58
1.13.0.dev20220924	0.3.0	0:52	0:36
	main	2:51	0:53
	m1-torch-1.13	2:33	0:54

As you can see, when attention slicing is enabled (using the pipeline's enable_attention_slicing() method), performance is better or much better, even though it's supposed to be slower. That's because when it's not in use this system requires more RAM than is physically installed and starts swapping, making everything extremely slow. I also observed in the activity monitor than swapping was indeed higher without attention slicing.

You are right in that PyTorch 1.13.0 + diffusers 0.3.0 hit a performance sweet spot. I haven't observed the same in my personal Mac running Ventura, so it could potentially be because of improvements to MPS in the new version, or it could be a combination of several factors.

I would recommend you enable attention slicing in your large image tests and see if that makes a difference.

@pcuenca

Hey @pcuenca, I almost did that yesterday since I was converting the FastAI course notebook you guys did for use on macOS — I mean enable attention slicing 🙂 I believe there was a cell which had attention slicing but it was commented out and I noticed that you'd mentioned attention slicing several times on the forum. But I ran into some other issue right at that point and forgot to check that!

Will run the whole suite of tests again today with attention slicing enabled and get you the results later today.

@pcuenca

@pcuenca Enabling attention slicing makes a huge difference! I'm putting up all the details again below both for historical purposes and for easier comparison but the all the generation times for any combination of size + torch + diffusers are now at much more acceptable levels after enabling attention slicing 🙂 I must admit that I didn't expect such a dramatic change in performance with that one change ...

Here's the full report:

macOS version is 12.6

RAM on device: 32GB

The height is 512 for all the following tests and the Python version is 3.9.13 and the warm up run always did 512 x 512 images but for 10 steps.

PyTorch versions used are 1.13.0.dev20220924 (before PyTorch image generation slowed down) and PyTorch 1.13.0 (pre RC).

Installed 1.13.0.dev20220924 using: pip install --upgrade --pre torch==1.13.0.dev20220924 --extra-index-url https://download.pytorch.org/whl/nightly/cpu

Installed 1.13.0 using: pip install --pre torch --extra-index-url https://download.pytorch.org/whl/test/cpu

Configuration	w = 512	w = 512 + attention slicing	w = 768	w = 768 + attention slicing
torch 1.13.0.dev20220924 diffusers 0.3.0	Warm up - 10s Full - 32.02s	Warm up - 7s Full - 32s	Warm up - 13s 56.57s	Warm up - 10s Full - 1m 5s
torch 1.13.0.dev20220924 diffusers 0.6.0	Warm up - 13s Full - 41s	Warm up - 12s Full - 43s	Warm up - 12s Full - 36m 45s	Warm up - 10s Full - 1m 45s
torch 1.13.0 diffusers 0.3.0	Warm up - 1m 40s Full - 6m 57s	Warm up - 7s Full - 34s	Warm up - 1m 37s Full - 30m 16s	Warm up - 8s Full - 58s
torch 1.13.0 diffusers 0.6.0	Warm up - 14s Full - 58s	Warm up - 10s Full - 43s	Warm up - 14s Full - 7m 37s	Warm up - 10s Full - 2m 6s

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

Navigation Menu

Repository navigation

Apple MPS image generation times take much longer with 0.4.0 #548

Describe the bug

Reproduction

Logs

System Info

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Apple MPS image generation times take much longer with 0.4.0 #548

Description

Describe the bug

Reproduction

Logs

System Info

Activity

patrickvonplaten commented on Oct 14, 2022

pcuenca commented on Oct 14, 2022

FahimF commented on Oct 15, 2022

FahimF commented on Oct 15, 2022

FahimF commented on Oct 17, 2022

pcuenca commented on Oct 18, 2022

FahimF commented on Oct 19, 2022

pcuenca commented on Oct 20, 2022

FahimF commented on Oct 20, 2022

pcuenca commented on Oct 20, 2022

FahimF commented on Oct 20, 2022

pcuenca commented on Oct 21, 2022

FahimF commented on Oct 21, 2022

pcuenca commented on Oct 21, 2022

FahimF commented on Oct 21, 2022

FahimF commented on Oct 22, 2022

github-actions commented on Nov 15, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions