clip-guided stable diffusion correctness

Describe the bug

the clip-guided pipeline uses these text-embeddings:

diffusers/examples/community/clip_guided_stable_diffusion.py

Lines 285 to 288 in 2345481

    
           # perform clip guidance 
        
           if clip_guidance_scale > 0: 
        
               text_embeddings_for_guidance = ( 
        
                   text_embeddings.chunk(2)[0] if do_classifier_free_guidance else text_embeddings

defined earlier as:

diffusers/examples/community/clip_guided_stable_diffusion.py

Lines 234 to 237 in 2345481

    
           # For classifier free guidance, we need to do two forward passes. 
        
           # Here we concatenate the unconditional and text embeddings into a single batch 
        
           # to avoid doing two forward passes 
        
           text_embeddings = torch.cat([uncond_embeddings, text_embeddings])

which I read as using the unconditioned (i.e. null prompt) embeddings.

Is that the way it's supposed to work? That doesn't feel like how it's supposed to work. Like, if the normal classifier-free guidance function is turned off, it would be using the embeddings from the text prompt, not the nulls.

But this pipeline was added without any tests or samples or other reference material, so I really don't know.

Reproduction

No response

Logs

No response

System Info

👀

@patil-suraj

cc @patil-suraj might be good to take a look soon since we just released it :-)

Someone posted another concern on discord:

I think there's a bug in the community CLIP guidance example -

diffusers/examples/community/clip_guided_stable_diffusion.py

Lines 132 to 135 in 8211b62

pred_original_sample = (latents - beta_prod_t ** (0.5) * noise_pred) / alpha_prod_t ** (0.5)

fac = torch.sqrt(beta_prod_t)

sample = pred_original_sample * (fac) + latents * (1 - fac)

IIUC the correct behavior would be to use pred_original_sample to do the CLIP comparison - i.e. it should be the sample, not pred_original_sample * (fac) + latents * (1 - fac)

@keturn

That's a great catch @keturn ! Sorry to come back to this only now, will send a fix right away.

Also updated the colab notebook https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/CLIP_Guided_Stable_diffusion_with_diffusers.ipynb#scrollTo=-bzXx-2EhlQY

Someone posted another concern on discord:

I think there's a bug in the community CLIP guidance example -

diffusers/examples/community/clip_guided_stable_diffusion.py

Lines 132 to 135 in 8211b62

pred_original_sample = (latents - beta_prod_t ** (0.5) * noise_pred) / alpha_prod_t ** (0.5)

fac = torch.sqrt(beta_prod_t)

sample = pred_original_sample * (fac) + latents * (1 - fac)

IIUC the correct behavior would be to use pred_original_sample to do the CLIP comparison - i.e. it should be the sample, not pred_original_sample * (fac) + latents * (1 - fac)

Afaik, this is correct because the conditional diffusion is conditional on predicted x_t rather than predicted x_0。

I think the correct format is "sample = pred_original_sample * (fac) + noise_pred * (1 - fac)"

Navigation Menu

Repository navigation

clip-guided stable diffusion correctness #596

Describe the bug

Reproduction

Logs

System Info

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# perform clip guidance
	if clip_guidance_scale > 0:
	text_embeddings_for_guidance = (
	text_embeddings.chunk(2)[0] if do_classifier_free_guidance else text_embeddings

	# For classifier free guidance, we need to do two forward passes.
	# Here we concatenate the unconditional and text embeddings into a single batch
	# to avoid doing two forward passes
	text_embeddings = torch.cat([uncond_embeddings, text_embeddings])

	pred_original_sample = (latents - beta_prod_t ** (0.5) * noise_pred) / alpha_prod_t ** (0.5)

	fac = torch.sqrt(beta_prod_t)
	sample = pred_original_sample * (fac) + latents * (1 - fac)

clip-guided stable diffusion correctness #596

Description

Describe the bug

Reproduction

Logs

System Info

Activity

patrickvonplaten commented on Sep 22, 2022

keturn commented on Sep 23, 2022

patil-suraj commented on Sep 28, 2022

patil-suraj commented on Sep 28, 2022

ajundo commented on Sep 29, 2022

zzhang2816 commented on Apr 11, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions