Repository navigation
refactors pipelines inheriting from StableDiffusionPipeline #6984
Description
We have a couple of community pipelines that used StableDiffusionPipeline as the base class instead of DiffusionPipeline. Most of them are broken after we changed the signature of StableDiffusionPipeline. see more details on the issue here #6969 (comment)
pipelines should not use StableDiffusionPipeline as a base class. Is anyone interested in refactoring some of these community pipelines to inherit from DiffusionPipeline instead? You will need to:
- update the base class to use
DiffusionPipelineand any of the mixins needed e.g.
class CommunityPipeline(
DiffusionPipeline, TextualInversionLoaderMixin, LoraLoaderMixin, IPAdapterMixin, FromSingleFileMixin
):- if the community pipeline use some of the methods in
StableDiffusionPipeline- you need use#Copied fromstatement to copy over methods to your community pipeline. Learn more about theCopied frommechanism here https://huggingface.co/docs/diffusers/conceptual/contribution#copied-from-mechanism
Here is the list of pipelines that need to be refactored!
- Prompt2PromptPipeline(completed by @ihkap11 Prompt2Prompt: Inherit from DiffusionPipeline #7211 ): https://github.com/huggingface/diffusers/blob/main/examples/community/pipeline_prompt2prompt.py
- RegionalPromptingStableDiffusionPipeline (@pranjalks is working on it): https://github.com/huggingface/diffusers/blob/main/examples/community/regional_prompting_stable_diffusion.py
- StableDiffusionReferencePipeline ( completed by @StandardAI [
Refactor]StableDiffusionReferencePipelineinheriting fromDiffusionPipeline#7071) - TensorRTStableDiffusionPipeline (@Bhavay-2001 is working on it): https://github.com/huggingface/diffusers/blob/main/examples/community/stable_diffusion_tensorrt_txt2img.py
Activity
pranjalks commentedon Feb 15, 2024
Hi @yiyixuxu, I am interested in resolving this issue. Please let me know if I can take this up.
tolgacangoz commentedon Feb 15, 2024
Thank you for this opportunity! I would like to take
StableDiffusionReferencePipeline.yiyixuxu commentedon Feb 15, 2024
hi @pranjalks
sure!
Just leave a message here about which ones you would like to work on:)
yiyixuxu commentedon Feb 15, 2024
@StandardAI
thanks!
pranjalks commentedon Feb 15, 2024
Hey, I have started working on
RegionalPromptingStableDiffusionPipeline. Thank you!nxbringr commentedon Feb 15, 2024
@yiyixuxu I'll work on Prompt2Prompt pipeline. Thank you!
Bhavay-2001 commentedon Feb 16, 2024
Hi @yiyixuxu, I would be happy to work TensorRTStableDiffusionPipeline.
Can you please assign it to me? Thanks
yiyixuxu commentedon Feb 16, 2024
@Bhavay-2001 sure!
katarzynasornat commentedon May 13, 2024
Hi @pranjalks is there any news on it? I have already used it and not working, wondering if it is finished, or should I also take a look and try to refactor it:)
yiyixuxu commentedon May 13, 2024
@katarzynasornat feel free to open a PR:)
jelling commentedon May 16, 2024
@yiyixuxu @katarzynasornat
I took a crack at upgrading the Regional Prompter but even with using the mix-ins I'm still getting this error:
Revised instantiation cribbed from #7071; I'm likely importing too much but nothing less worked either.
Python isn't my main language though so perhaps I'm doing something dumb or underestimating the amount of refactoring needed.
katarzynasornat commentedon May 17, 2024
@jelling @yiyixuxu I will try to help to refactor it but before doing that (as I am quite new to diffusers and not a pro in Deep Learning but I am very fast student:)) I wanted to downgrade to a lower version of diffusers - before the change happened. Unfortunately I had another error which made me impossible to run and see if/how the pipeline works.
Below I enclosed my gist. Would you be kind to take a look what could go wrong? Otherwise hard to improve the pipeline which may not working anyway.
tolgacangoz commentedon May 17, 2024
@jelling Tip: 2 parameters were expected additionally, right? You can see something about these 2 parameters in my PR.
katarzynasornat commentedon May 18, 2024
@StandardAI Are you able to look at my gist after downgrading diffusers? Because after refactoring I am getting the same error and I am not sure if refactoring went wrong or the pipeline is somehow not working itself.
Thank you!
EDIT: error looks somehow like this
RuntimeError Traceback (most recent call last)
in <cell line: 1>()
----> 1 images = pipe(
2 prompt=prompt,
3 negative_prompt="""ugly""",
4 guidance_scale=7.5,
5 height = 512,
18 frames
in forward(hidden_states, encoder_hidden_states, attention_mask, temb, scale)
512 head_dim = inner_dim // attn.heads
513
--> 514 query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
515
516 key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
RuntimeError: shape '[5, -1, 8, 40]' is invalid for input of size 5242880
tolgacangoz commentedon May 26, 2024
Debugging for a previous version of
diffusersmight be unproductive, IMHO. If you open a refactoring PR I can guide you. You can take a look at the previously merged refactorings within this issue.katarzynasornat commentedon May 28, 2024
@tolgacangoz so cool! Will open a PR today. Assume I should work on current
diffusersversion 0.28.0.dev0? I took a look at the previous examples, what I found is that all of them (I think) where usingStableDiffusionPipelineOutput, but regional prompt is invokingStableDiffusionPipelineinside. Happy to work and finish it properly with your guidance!a-r-r-o-w commentedon Jul 15, 2025
AFAICT, all the tasks here are completed, so marking this as resolved. Thanks everyone for your contributions 🤗