Adding logging images to textual inversion. Issue 1837

[isamu-isozaki]

This is the implementation of the idea of this issue
I added support for wandb if it is installed but will otherwise just log the images to args.output_dir/imgs/

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[patrickvonplaten]

@patil-suraj @williamberman could you take a look here?

[isamu-isozaki]

sry, I'll fix the quality check in a bit

[patrickvonplaten]

Re-ping @patil-suraj @williamberman here

[isamu-isozaki]

Quality checks for the files inside examples/textual_inversion passed so I'm done with what I wanted to do I think.

[patrickvonplaten]

@patil-suraj can you check please?

[patil-suraj]

Thanks a lot for the PR! Sorry to only come back to it now. The logic can be simplified a lot, for reference see

diffusers/examples/dreambooth/train_dreambooth_lora.py

Lines 910 to 944 in a66f2ba

	if args.validation_prompt is not None and epoch % args.validation_epochs == 0:
	logger.info(
	f"Running validation... \n Generating {args.num_validation_images} images with prompt:"
	f" {args.validation_prompt}."
	)
	# create pipeline
	pipeline = DiffusionPipeline.from_pretrained(
	args.pretrained_model_name_or_path,
	unet=accelerator.unwrap_model(unet),
	text_encoder=accelerator.unwrap_model(text_encoder),
	revision=args.revision,
	torch_dtype=weight_dtype,
	)
	pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
	pipeline = pipeline.to(accelerator.device)
	pipeline.set_progress_bar_config(disable=True)
	# run inference
	generator = torch.Generator(device=accelerator.device).manual_seed(args.seed)
	prompt = args.num_validation_images * [args.validation_prompt]
	images = pipeline(prompt, num_inference_steps=25, generator=generator).images
	for tracker in accelerator.trackers:
	if tracker.name == "wandb":
	tracker.log(
	{
	"validation": [
	wandb.Image(image, caption=f"{i}: {args.validation_prompt}")
	for i, image in enumerate(images)
	]
	}
	)
	del pipeline
	torch.cuda.empty_cache()

Could you please update the PR with a similar logic as above. Thanks :)

[isamu-isozaki]

@patil-suraj Thanks for the review! And will do

[isamu-isozaki]

@patil-suraj Should be fixed now! Tested with command

accelerate launch textual_inversion.py --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4" --train_data_dir='frida' --learnable_property="object" --placeholder_token="<frida>" --initializer_token="dog" --resolution=512 --train_batch_size=1 --gradient_accumulation_steps=4 --max_train_steps=3000 --learning_rate=5.0e-04 --scale_lr --lr_scheduler="constant" --lr_warmup_steps=0 --output_dir="textual_inversion_frida" --report_to="wandb" --validation_prompt="A photo of <frida>" --log_frequency=1 --revision="fp16" --mixed_precision="fp16" --enable_xformers_memory_efficient_attention

I needed to modify 1 line of StableDiffusionPipeline so that text_encoder's dtype is fine even with mixed_precision