CLIPSeg + StableDiffusionInpainting

@NielsRogge

Describe the bug

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-10-1ff2265a51a8>](https://localhost:8080/#) in <cell line: 7>()
      5 # model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined")
      6 
----> 7 pipe = load_sd_model()

1 frames
[/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/pipeline_utils.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
   1139         elif len(missing_modules) > 0:
   1140             passed_modules = set(list(init_kwargs.keys()) + list(passed_class_obj.keys())) - optional_kwargs
-> 1141             raise ValueError(
   1142                 f"Pipeline {pipeline_class} expected {expected_modules}, but only {passed_modules} were passed."
   1143             )

ValueError: Pipeline <class 'diffusers_modules.git.text_inpainting.TextInpainting'> expected {'safety_checker', 'tokenizer', 'segmentation_processor', 'vae', 'segmentation_model', 'feature_extractor', 'text_encoder', 'unet', 'scheduler'}, but only {'tokenizer', 'segmentation_processor', 'vae', 'segmentation_model', 'feature_extractor', 'text_encoder', 'unet', 'scheduler'} were passed.

This issue was mentioned #1250 but does not look like it still works
Whereas this example is shared :

https://github.com/huggingface/diffusers/blob/main/examples/community/README.md by @NielsRogge , i just changed the model since runwayml/stable-diffusion-inpainting shows "unet\diffusion_pytorch_model.safetensors not found" as also mentioned in #3102

Reproduction

I tried something similar but got the above error:

pip install -q xformers==0.0.20 triton==2.0.0 gradio_client==0.2.7 -U
pip install -q torch==2.0.1+cu118 torchvision==0.15.2+cu118 torchaudio==2.0.2+cu118 torchtext==0.15.2 torchdata==0.6.1 --extra-index-url https://download.pytorch.org/whl/cu118 -U
pip install -q 'git+https://github.com/huggingface/diffusers'
pip install -Uq 'git+https://github.com/huggingface/transformers'

from transformers import CLIPSegProcessor, CLIPSegForImageSegmentation


from diffusers import StableDiffusionPipeline, StableDiffusionInpaintPipeline, StableDiffusionInstructPix2PixPipeline,DiffusionPipeline
from diffusers import EulerAncestralDiscreteScheduler,DPMSolverMultistepScheduler,UniPCMultistepScheduler


output_dir = "outputs" #@param
device = "cuda" if torch.cuda.is_available() else "cpu"
url = "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/timojl/clipseg/blob/master/example_image.jpg?raw=true" #@param
image = Image.open(requests.get(url, stream=True).raw).resize((512, 512))
text = "a glass"  #@param will mask out this text
prompt = "a cup"  #@param the masked out region will be replaced with this


# examples = [["example_image.png", "a glass", "a cup"]]

os.makedirs(output_dir, exist_ok=True)


sd_pipe = StableDiffusionInpaintPipeline.from_pretrained(
                "stabilityai/stable-diffusion-2-inpainting",
                custom_pipeline="text_inpainting",
                segmentation_model=CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined"),
                segmentation_processor=CLIPSegProcessor.from_pretrained("CIDAS/clipseg-rd64-refined"),
                use_auth_token=auth_token,
                torch_dtype=torch.float16
        ).to(device)
        sd_pipe.enable_xformers_memory_efficient_attention()
        # change scheduler
        sd_pipe.scheduler = UniPCMultistepScheduler.from_config(sd_pipe.scheduler.config)

inpainted_image = pipe(image=image, text=text, prompt=prompt).images[0]

Logs

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-1ff2265a51a8> in <cell line: 7>()
      5 # model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined")
      6 
----> 7 pipe = load_sd_model()

1 frames
<ipython-input-8-d4db719ed3c1> in load_sd_model()
      5     sd_pipe = None
      6     if os.environ.get('IS_MY_DEBUG') is None:
----> 7         sd_pipe = StableDiffusionInpaintPipeline.from_pretrained(
      8                 "stabilityai/stable-diffusion-2-inpainting",
      9                 custom_pipeline="text_inpainting",

/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/pipeline_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
   1139         elif len(missing_modules) > 0:
   1140             passed_modules = set(list(init_kwargs.keys()) + list(passed_class_obj.keys())) - optional_kwargs
-> 1141             raise ValueError(
   1142                 f"Pipeline {pipeline_class} expected {expected_modules}, but only {passed_modules} were passed."
   1143             )

ValueError: Pipeline <class 'diffusers_modules.git.text_inpainting.TextInpainting'> expected {'safety_checker', 'tokenizer', 'segmentation_processor', 'vae', 'segmentation_model', 'feature_extractor', 'text_encoder', 'unet', 'scheduler'}, but only {'tokenizer', 'segmentation_processor', 'vae', 'segmentation_model', 'feature_extractor', 'text_encoder', 'unet', 'scheduler'} were passed.



### System Info

colab notebook

### Who can help?

@NielsRogge @patrickvonplaten @sayakpaul

You cannot pass:

 segmentation_model=CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined"),
                segmentation_processor=CLIPSegProcessor.from_pretrained("CIDAS/clipseg-rd64-refined"),

to an inpainting pipeline

@patrickvonplaten

You cannot pass:

 segmentation_model=CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined"),
                segmentation_processor=CLIPSegProcessor.from_pretrained("CIDAS/clipseg-rd64-refined"),

to an inpainting pipeline

Hi @patrickvonplaten ,
Thanks for your reply, i got from your git repo(Note: Text Based Inpainting Stable Diffusion(https://github.com/huggingface/diffusers/tree/main/examples/community)
There is a HF space : https://huggingface.co/spaces/nielsr/text-based-inpainting/blob/main/app.py



from transformers import CLIPSegProcessor, CLIPSegForImageSegmentation
from diffusers import DiffusionPipeline

from PIL import Image
import requests

processor = CLIPSegProcessor.from_pretrained("CIDAS/clipseg-rd64-refined")
model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined")

pipe = DiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    custom_pipeline="text_inpainting",
    segmentation_model=model,
    segmentation_processor=processor
)
pipe = pipe.to("cuda")


url = "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/timojl/clipseg/blob/master/example_image.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw).resize((512, 512))
text = "a glass"  # will mask out this text
prompt = "a cup"  # the masked out region will be replaced with this

image = pipe(image=image, text=text, prompt=prompt).images[0]

Looking forward to hearing from you.
Thanks,

@andysingal

@andysingal you could try to run it yourself, e.g.

from diffusers import StableDiffusionInpaintPipeline
from transformers import AutoProcessor, CLIPSegForImageSegmentation
from PIL import Image
import requests
import torch

processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined")
model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined")
pipe = StableDiffusionInpaintPipeline.from_pretrained("stabilityai/stable-diffusion-2-inpainting")
pipe.to("cuda")

url = "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/timojl/clipseg/blob/master/example_image.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw).resize((512, 512))
text = "a glass"  # will mask out this text
prompt = "a cup"  # the masked out region will be replaced with this

inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt")
outputs = model(**inputs)
mask = torch.sigmoid(outputs.logits).cpu().detach().numpy() 
mask_pil = Image.fromarray(mask * 255)
image = pipe(prompt=[prompt], image=image, mask_image=mask_pil).images[0]

@andysingal

@andysingal you could try to run it yourself, e.g.

from diffusers import StableDiffusionInpaintPipeline
from transformers import AutoProcessor, CLIPSegForImageSegmentation
from PIL import Image
import requests
import torch

processor = AutoProcessor.from_pretrained("CIDAS/clipseg-rd64-refined")
model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined")
pipe = StableDiffusionInpaintPipeline.from_pretrained("stabilityai/stable-diffusion-2-inpainting")
pipe.to("cuda")

url = "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/timojl/clipseg/blob/master/example_image.jpg?raw=true"
image = Image.open(requests.get(url, stream=True).raw).resize((512, 512))
text = "a glass"  # will mask out this text
prompt = "a cup"  # the masked out region will be replaced with this

inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt")
outputs = model(**inputs)
mask = torch.sigmoid(outputs.logits).cpu().detach().numpy() 
mask_pil = Image.fromarray(mask * 255)
image = pipe(prompt=[prompt], image=image, mask_image=mask_pil).images[0]

Hi @DaniHi , sorry for the late reply i will try it out. I am working on the alternative GroundingDINO + SAM + inpaint as part of my work research. Thanks Again!!

Navigation Menu

Repository navigation

CLIPSeg + StableDiffusionInpainting #4464

Describe the bug

Reproduction

Logs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CLIPSeg + StableDiffusionInpainting #4464

Description

Describe the bug

Reproduction

Logs

Activity

patrickvonplaten commented on Aug 4, 2023

andysingal commented on Aug 4, 2023

DaniHi commented on Aug 5, 2023

andysingal commented on Aug 9, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions