is it necessary to add a description of each picture in the training set？

[Jason-kid]

Hey, thanks for this great project, I'm a bit confused.
For painting style learning, is it necessary to add a description of each picture in the training set? For example, 1) use a unique word to represent the style of painting, such as used in dreambooth lora, and there is no image description file; 2) use image-text pair data, such as in train_txt_image_lora, a picture corresponds to a description file.
What is the difference between them? Which method is more conducive to painting style learning and reasoning use?

[patrickvonplaten]

@sayakpaul could you check here?

[sayakpaul]

Hey there! Thanks for your kind words and your question. I believe it greatly serves the purpose of open-science :)

1. use a unique word to represent the style of painting, such as used in dreambooth lora, and there is no image description file

Well, for DreamBooth the captions / description / prompts actually stay the same for all the images used for DreamBooth training. To make things concrete, here's a snapshot of the dataset we use in the DreamBooth training example:

The [instance_id] (shown in the image above) we use in the example is "sks". As you can see, the prompts are one and the same and in the example, we refer to this prompt as instance_prompt:

diffusers/examples/dreambooth/train_dreambooth.py

Line 169 in bbab855

"--instance_prompt",

Along with this dataset, when prior preservation is enabled, we make use of the class_prompt:

diffusers/examples/dreambooth/train_dreambooth.py

Line 176 in bbab855

"--class_prompt",

Even when using prior preservation, the same class_prompt is used for generating the "class images".

In a sense, these datasets are still paired, i.e., an image is paired with some description / caption / prompt but those captions are one and the same. Whereas, in the vanilla text-to-image case, the dataset has image-caption pairs where the captions are different for each image. Like this:

I hope the difference between the datasets is now clear.

[sayakpaul]

Coming to

What is the difference between them? Which method is more conducive to painting style learning and reasoning use?

I believe the primary difference lies in their objectives and scopes. DreamBooth is more targeted toward customizing the generation toward a very specific visual concept (like a specific type of face or any other object).

But for the vanilla text-to-image fine-tuning, we are interested in being a bit more general toward the generation process. In our example, we're interested to fine-tune the model such that the generated images appear in the visual context of Pokemons. We don't need to rely on using the instance identifier (like "sks") in our inference prompts here as opposed to DreamBooth.

So, to answer your question, if you're aiming for generality, then I'd suggest going for regular text-to-image fine-tuning. We're working on InstructPix2Pix (#2478) that might interest you. If you want the generation to be ultra-specific, I'd suggest going for DreamBooth or even Textual Inversion.

I hope it answers your question :-)

[Jason-kid]

Thanks for your patience in answering, i get it and close the issue.

[edwardelric1202]

Hey there! Thanks for your kind words and your question. I believe it greatly serves the purpose of open-science :)

1. use a unique word to represent the style of painting, such as used in dreambooth lora, and there is no image description file

Well, for DreamBooth the captions / description / prompts actually stay the same for all the images used for DreamBooth training. To make things concrete, here's a snapshot of the dataset we use in the DreamBooth training example:

The [instance_id] (shown in the image above) we use in the example is "sks". As you can see, the prompts are one and the same and in the example, we refer to this prompt as instance_prompt:

diffusers/examples/dreambooth/train_dreambooth.py

Line 169 in bbab855

"--instance_prompt",

Along with this dataset, when prior preservation is enabled, we make use of the class_prompt:

diffusers/examples/dreambooth/train_dreambooth.py

Line 176 in bbab855

"--class_prompt",

Even when using prior preservation, the same class_prompt is used for generating the "class images".

In a sense, these datasets are still paired, i.e., an image is paired with some description / caption / prompt but those captions are one and the same. Whereas, in the vanilla text-to-image case, the dataset has image-caption pairs where the captions are different for each image. Like this:

I hope the difference between the datasets is now clear.

Hi, I wonder if I can train multiple concepts in once dreambooth training. For example, I got some images of dogs and cats in pixelated style, and I want train a model that can generate like "a pixelated style cat" and "a pixelated dog", is it possible for diffusers now?

[sayakpaul]

No, we currently don't support DreamBoothing with multiple concepts.

[edwardelric1202]

No, we currently don't support DreamBoothing with multiple concepts.

Ok, thanks for your reply.

[Adenialzz]

Hey there! Thanks for your kind words and your question. I believe it greatly serves the purpose of open-science :)

1. use a unique word to represent the style of painting, such as used in dreambooth lora, and there is no image description file

Well, for DreamBooth the captions / description / prompts actually stay the same for all the images used for DreamBooth training. To make things concrete, here's a snapshot of the dataset we use in the DreamBooth training example:

The [instance_id] (shown in the image above) we use in the example is "sks". As you can see, the prompts are one and the same and in the example, we refer to this prompt as instance_prompt:

diffusers/examples/dreambooth/train_dreambooth.py

Line 169 in bbab855

"--instance_prompt",

Along with this dataset, when prior preservation is enabled, we make use of the class_prompt:

diffusers/examples/dreambooth/train_dreambooth.py

Line 176 in bbab855

"--class_prompt",

Even when using prior preservation, the same class_prompt is used for generating the "class images".

In a sense, these datasets are still paired, i.e., an image is paired with some description / caption / prompt but those captions are one and the same. Whereas, in the vanilla text-to-image case, the dataset has image-caption pairs where the captions are different for each image. Like this:

I hope the difference between the datasets is now clear.

Hi, thanks for your explanation. But when we train dreambooth about specific style, we may want to attach semantic captions describing the image content along with the style keyword, like: a girl standing on the street, sks style. So how can we adapt DreamBoothDataset to text-image-pair format ?