[textual inversion] Allow validation images

[pcuenca]

No description provided.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Thanks a lot!

[pcuenca]

Oops, I wrote this yesterday while I was working on a different textual inversion fix, but did not realize #1838 existed. @patil-suraj we can close this one and keep working on #1838 as it was first, or we can add @isamu-isozaki as an author here, what do you think?

[isamu-isozaki]

@pcuenca Personally happy with either! I think there's a slight bug when using mixed_precision=fp16 because the text encoder will be in fp32 and everything else is in fp16. Also, probably assigning unet=unet, vae=vae in the pipeline is better because it uses less memory. For my gpu, I did keep getting oom when I didn't do that.

[isamu-isozaki]

Also, very minor but logging/validation with global_step might be cleaner since we are doing that for save anyways

[patil-suraj]

Looks good to me! Think we can add isamu-isozaki as co-author and merge this PR.

[pcuenca]

@pcuenca Personally happy with either! I think there's a slight bug when using mixed_precision=fp16 because the text encoder will be in fp32 and everything else is in fp16.

I think that's only true if we reuse the unet and vae like you suggest. Otherwise, from_pretrained will load everything in float32, as this comment explains: https://github.com/huggingface/diffusers/pull/2077/files#diff-a3d54a67360d6c7648a2de57fea2d72444395d0ba56eb4ba8545c6ca820c3f47R776

Also, probably assigning unet=unet, vae=vae in the pipeline is better because it uses less memory. For my gpu, I did keep getting oom when I didn't do that.

Yeah, in that case we'd need to load the pipeline setting torch_dtype to the half precision type.

I'm hesitant about doing this here because all the other training scripts use the same approach, and in general diffusers defaults to running inference using the highest precision. Maybe open a new PR to discuss the implications, or make it an optional argument for validation?

[pcuenca]

Also, very minor but logging/validation with global_step might be cleaner since we are doing that for save anyways

I do agree with this, it's weird to me that the frequency of some tasks is determined in epochs while others are expressed in steps. Would you be willing to open an issue/PR to make it consistent in all training scripts?

[pcuenca]

Merging as is, but those are some valuable comments, @isamu-isozaki! Let's try to deal with them in new PRs so we can discuss and apply any changes to all training scripts :) Thanks a lot! 🙌

[isamu-isozaki]

@pcuenca Thanks for the comments! Will make new issues for both to discuss