Repository navigation

Add UniDiffuser model and pipeline - #2963

Merged

sayakpaul merged 334 commits into

huggingface:mainfrom

dg845:unidiffuser-pipeline

May 26, 2023

This pull request is big! We're only showing the most recent 250 commits

Commits on May 5, 2023

Fix a bug of pano when not doing CFG (#3030)

authored and
dg845
committed
115e382
Text2video zero refinements (#3070)

19and99
authored and
dg845
committed
10c54cb
Release: v0.15.0

patrickvonplaten
authored and
dg845
committed
945f300
[Tests] Speed up panorama tests (#3067)

sayakpaul
authored and
dg845
committed
322b5cb
[Post release] v0.16.0dev (#3072)

patrickvonplaten
authored and
dg845
committed
af0c3a7
Adds profiling flags, computes train metrics average. (#3053)

authored and
dg845
committed
7a1d100
[Pipelines] Make sure that None functions are correctly not saved (#3080)

patrickvonplaten
authored and
dg845
committed
bbabf3f
doc string example remove from_pt (#3083)

yiyixuxu
authored and
dg845
committed
068d6b4
[Tests] parallelize (#3078)

authored and
dg845
committed
9dd6058
Throw deprecation warning for return_cached_folder (#3092)

patrickvonplaten
authored and
dg845
committed
74907ee
Allow SD attend and excite pipeline to work with any size output images (#2835)

jcoffland
authored and
dg845
committed
0be9f8b
[docs] Update community pipeline docs (#2989)

stevhliu
authored and
dg845
committed
d6ae0ae
Add to support Guess Mode for StableDiffusionControlnetPipleline (#2998)

takuma104
authored and
dg845
committed
7f3cb6d
fix default value for attend-and-excite (#3099)

yiyixuxu
authored and
dg845
committed
1184b36
remvoe one line as requested by gc team (#3077)

yiyixuxu
authored and
dg845
committed
fa6a6b4
ddpm custom timesteps (#3007)

williamberman
authored and
dg845
committed
a256f84
Fix breaking change in `pipeline_stable_diffusion_controlnet.py` (#3118)

remorses
authored and
dg845
committed
e9cb03e
Add global pooling to controlnet (#3121)

patrickvonplaten
authored and
dg845
committed
c98e41d
[Bug fix] Fix img2img processor with safety checker (#3127)

patrickvonplaten
authored and
dg845
committed
653b3c1
[Bug fix] Make sure correct timesteps are chosen for img2img (#3128)

patrickvonplaten
authored and
dg845
committed
7fa3b6c
Improve deprecation warnings (#3131)

patrickvonplaten
authored and
dg845
committed
cc14690
Fix config deprecation (#3129)

authored and
dg845
committed
6f12a36
feat: verfication of multi-gpu support for select examples. (#3126)

sayakpaul
authored and
dg845
committed
16ddd8b
speed up attend-and-excite fast tests (#3079)

yiyixuxu
authored and
dg845
committed
07731e9
Optimize log_validation in train_controlnet_flax (#3110)

cgarciae
authored and
dg845
committed
7a39b0f
make style

patrickvonplaten
authored and
dg845
committed
7ae597f
Correct textual inversion readme (#3145)

patrickvonplaten
authored and
dg845
committed
00a5e55
Add unet act fn to other model components (#3136)

williamberman
authored and
dg845
committed
ff5b99b
class labels timestep embeddings projection dtype cast (#3137)

williamberman
authored and
dg845
committed
c8eaea5
[ckpt loader] Allow loading the Inpaint and Img2Img pipelines, while loading a ckpt model (#2705)

cmdr2
authored and
dg845
committed
1fac211
add from_ckpt method as Mixin (#2318)

authored and
dg845
committed
f3300a8
Add TensorRT SD/txt2img Community Pipeline to diffusers along with TensorRT utils (#2974)

authored and
dg845
committed
fc3760d
Correct `Transformer2DModel.forward` docstring (#3074)

offchan42
authored and
dg845
committed
6058378
Update pipeline_stable_diffusion_inpaint_legacy.py (#2903)

authored and
dg845
committed
e5335f3
Modified altdiffusion pipline to support altdiffusion-m18 (#2993)

authored and
dg845
committed
4afb911
controlnet training resize inputs to multiple of 8 (#3135)

williamberman
authored and
dg845
committed
b320c6b
adding custom diffusion training to diffusers examples (#3031)

authored and
dg845
committed
3417b1f
make style

patrickvonplaten
authored and
dg845
committed
4a6aee9
Update custom_diffusion.mdx (#3165)

Mishig
authored and
dg845
committed
3306b04
Added distillation for quantization example on textual inversion. (#2760)

XinyuYe-Intel
authored and
dg845
committed
abd21da
Update Noise Autocorrelation Loss Function for Pix2PixZero Pipeline (#2942)

clarencechen
authored and
dg845
committed
f99a9ff
[DreamBooth] add text encoder LoRA support in the DreamBooth training script (#3130)

sayakpaul
authored and
dg845
committed
206b9b6
Update Habana Gaudi documentation (#3169)

regisss
authored and
dg845
committed
6ad4392
Add model offload to x4 upscaler (#3187)

patrickvonplaten
authored and
dg845
committed
4366b0c
[docs] Deterministic algorithms (#3172)

stevhliu
authored and
dg845
committed
e21784e
Update custom_diffusion.mdx to credit the author (#3163)

sayakpaul
authored and
dg845
committed
006ae03
Fix TensorRT community pipeline device set function (#3157)

authored and
dg845
committed
dac4d4a
make `from_flax` work for controlnet (#3161)

authored and
dg845
committed
c98a055
[docs] Clarify training args (#3146)

stevhliu
authored and
dg845
committed
15a90e2
Multi Vector Textual Inversion (#3144)

authored and
dg845
committed
cf35763
Add `Karras sigmas` to HeunDiscreteScheduler (#3160)

authored and
dg845
committed
c729403
[AudioLDM] Fix dtype of returned waveform (#3189)

sanchit-gandhi
authored and
dg845
committed
43c90b0
Fix bug in train_dreambooth_lora (#3183)

crywang
authored and
dg845
committed
49c9b4c
[Community Pipelines] Update lpw_stable_diffusion pipeline (#3197)

SkyTNT
authored and
dg845
committed
a69502f
Make sure VAE attention works with Torch 2_0 (#3200)

patrickvonplaten
authored and
dg845
committed
3267649
Revert "[Community Pipelines] Update lpw_stable_diffusion pipeline" (#3201)

williamberman
authored and
dg845
committed
de05ea0
[Bug fix] Fix batch size attention head size mismatch (#3214)

patrickvonplaten
authored and
dg845
committed
8953209
fix mixed precision training on train_dreambooth_inpaint_lora (#3138)

themrzmaster
authored and
dg845
committed
4e03663
adding enable_vae_tiling and disable_vae_tiling functions (#3225)

init-22
authored and
dg845
committed
167cb7a
Add ControlNet v1.1 docs (#3226)

patrickvonplaten
authored and
dg845
committed
0431637
Fix issue in maybe_convert_prompt (#3188)

pdoane
authored and
dg845
committed
9e2f445
Sync cache version check from transformers (#3179)

ychfan
authored and
dg845
committed
81950af
Fix docs text inversion (#3166)

patrickvonplaten
authored and
dg845
committed
711119a
add model (#3230)

authored and
dg845
committed
416f31a
Allow return pt x4 (#3236)

patrickvonplaten
authored and
dg845
committed
7ad77dd
Allow fp16 attn for x4 upscaler (#3239)

patrickvonplaten
authored and
dg845
committed
3acc879
fix fast test (#3241)

patrickvonplaten
authored and
dg845
committed
4c73947
Adds a document on token merging (#3208)

sayakpaul
authored and
dg845
committed
81d7eba
[AudioLDM] Update docs to use updated ckpt (#3240)

sanchit-gandhi
authored and
dg845
committed
f83fbbd
Release: v0.16.0

patrickvonplaten
authored and
dg845
committed
4cc60b5
Post release for 0.16.0 (#3244)

patrickvonplaten
authored and
dg845
committed
163c33b
[docs] only mention one stage (#3246)

authored and
dg845
committed
59986b6
Write model card in controlnet training script (#3229)

pcuenca
authored and
dg845
committed
a640f1b
[2064]: Add stochastic sampler (sample_dpmpp_sde) (#3020)

authored and
dg845
committed
7880ed7
[Stochastic Sampler][Slow Test]: Cuda test fixes (#3257)

authored and
dg845
committed
8def721
Remove required from tracker_project_name (#3260)

pcuenca
authored and
dg845
committed
97cf386
adding required parameters while calling the get_up_block and get_down_block (#3210)

authored and
dg845
committed
cf2bf70
[docs] Update interface in repaint.mdx (#3119)

ernestchu
authored and
dg845
committed
76e5941
Update IF name to XL (#3262)

authored and
dg845
committed
1147c76
fix typo in score sde pipeline (#3132)

fecet
authored and
dg845
committed
cd13b10
Fix typo in textual inversion JAX training script (#3123)

jairtrejo
authored and
dg845
committed
5f3b10a
AudioDiffusionPipeline - fix encode method after config changes (#3114)

teticio
authored and
dg845
committed
9585b23
Revert "Revert "[Community Pipelines] Update lpw_stable_diffusion pipeline"" (#3265)

patrickvonplaten
authored and
dg845
committed
79706a7
Fix community pipelines (#3266)

patrickvonplaten
authored and
dg845
committed
12868b1
update notebook (#3259)

authored and
dg845
committed
2c87f65
[docs] add notes for stateful model changes (#3252)

authored and
dg845
committed
a80f696
[LoRA] quality of life improvements in the loading semantics and docs (#3180)

authored and
dg845
committed
72a8467
[Community Pipelines] EDICT pipeline implementation (#3153)

authored and
dg845
committed
716c255
[Docs]zh translated docs update (#3245)

DrDavidS
authored and
dg845
committed
c1dce20
Update logging.mdx (#2863)

tolgacangoz
authored and
dg845
committed
5151f21
Add multiple conditions to StableDiffusionControlNetInpaintPipeline (#3125)

timegate
authored and
dg845
committed
ffe6e92
Let's make sure that dreambooth always uploads to the Hub (#3272)

patrickvonplaten
authored and
dg845
committed
10d856a
Diffedit Zero-Shot Inpainting Pipeline (#2837)

clarencechen
authored and
dg845
committed
029a28f
add constant learning rate with custom rule (#3133)

jason9075
authored and
dg845
committed
fd47d7c
Allow disabling torch 2_0 attention (#3273)

patrickvonplaten
authored and
dg845
committed
220657b
[doc] add link to training script (#3271)

authored and
dg845
committed
1b95720
temp disable spectogram diffusion tests (#3278)

williamberman
authored and
dg845
committed
08fbaaf
Changed sample[0] to images[0] (#3304)

IliaLarchenko
authored and
dg845
committed
799015c
Typo in tutorial (#3295)

IliaLarchenko
authored and
dg845
committed
c38d004
Torch compile graph fix (#3286)

patrickvonplaten
authored and
dg845
committed
6a84a74
Postprocessing refactor img2img (#3268)

authored and
dg845
committed
863bb75
[Torch 2.0 compile] Fix more torch compile breaks (#3313)

authored and
dg845
committed
c8cc4f0
fix: scale_lr and sync example readme and docs. (#3299)

sayakpaul
authored and
dg845
committed
6e8d065
Update stable_diffusion.mdx (#3310)

mu94-csl
authored and
dg845
committed
d38b4d9
Fix missing variable assign in DeepFloyd-IF-II (#3315)

gitmylo
authored and
dg845
committed
0d22064
Correct doc build for patch releases (#3316)

patrickvonplaten
authored and
dg845
committed
5a75a8a
Add Stable Diffusion RePaint to community pipelines (#3320)

Markus-Pobitzer
authored and
dg845
committed
5ea3424
Fix multistep dpmsolver for cosine schedule (suitable for deepfloyd-if) (#3314)

authored and
dg845
committed
7815c41
[docs] Improve LoRA docs (#3311)

stevhliu
authored and
dg845
committed
0e8f4f0
Added input pretubation (#3292)

isamu-isozaki
authored and
dg845
committed
7929587
Update write_own_pipeline.mdx (#3323)

csaybar
authored and
dg845
committed
3e8d3d8
update controlling generation doc with latest goodies. (#3321)

sayakpaul
authored and
dg845
committed
b4aa419
[Quality] Make style (#3341)

patrickvonplaten
authored and
dg845
committed
458847e
Fix config dpm (#3343)

patrickvonplaten
authored and
dg845
committed
140ab74
Add the SDE variant of DPM-Solver and DPM-Solver++ (#3344)

LuChengTHU
authored and
dg845
committed
652dbaa
Add upsample_size to AttnUpBlock2D, AttnDownBlock2D (#3275)

Will Rice
authored and
dg845
committed
1d213de
Add UniDiffuser classes to __init__ files, modify transformer block to support pre- and post-LN, add fast default tests, fix some bugs.
dg845
committed
434b255
Update fast tests to use test checkpoints stored on the hub and to better match the reference UniDiffuser implementation.
dg845
committed
7097dd7
Fix code with make style.
dg845
committed
fc85263
Revert "Fix code style with make style."
dg845
committed
9d39bef
Merge branch 'main' into unidiffuser-pipeline
dg845
committed
1cb726a
Add self.image_encoder, self.text_decoder to list of models to offload to CPU in the enable_sequential_cpu_offload(...)/enable_model_cpu_offload(...) methods to make test_cpu_offload_forward_pass p…
dg845
committed
e62b32a
Fix code quality with make style.
dg845
committed
fc540b5

Commits on May 6, 2023

Support using a data type embedding for UniDiffuser-v1.
dg845
committed
54c495f

Commits on May 9, 2023

Commits on May 11, 2023

Commits on May 12, 2023

Fix shape errors for the 'joint' and 'img2text' modes.
dg845
committed
4d656b5

Commits on May 15, 2023

Commits on May 16, 2023

Improve UniDiffuser docs, particularly the usage examples, and improve slow tests with new expected outputs.
dg845
committed
e56fab2
make style
dg845
committed
ecaf07f
Fix examples to load model in float16.
dg845
committed
c161e29

Commits on May 17, 2023

In image-to-text mode, sample from the autoencoder moment distribution instead of always getting its mode.
dg845
committed
926c7fb
make style
dg845
committed
edbadcc

Commits on May 21, 2023

When encoding the image using the VAE, scale the image latents by the VAE's scaling factor.
dg845
committed
6b35c03
make style
dg845
committed
f46593e
Clean up code and make slow tests pass.
dg845
committed
ec7fb87
make fix-copies

patrickvonplaten
authored and
dg845
committed
029c96c
[docs] Fix docstring (#3334)

authored and
dg845
committed
6644d11
if dreambooth lora (#3360)

williamberman
authored and
dg845
committed
c221086
Postprocessing refactor all others (#3337)

authored and
dg845
committed
f670e08
[docs] Improve safetensors docstring (#3368)

stevhliu
authored and
dg845
committed
7266fc1
add: a warning message when using xformers in a PT 2.0 env. (#3365)

authored and
dg845
committed
4b76097
StableDiffusionInpaintingPipeline - resize image w.r.t height and width (#3322)

authored and
dg845
committed
6e297b4
make style

patrickvonplaten
authored and
dg845
committed
fec7bd1
[docs] Adapt a model (#3326)

stevhliu
authored and
dg845
committed
e162d49
[docs] Load safetensors (#3333)

authored and
dg845
committed
caa080c
make style

patrickvonplaten
authored and
dg845
committed
75c2f75
[Docs] Fix stable_diffusion.mdx typo (#3398)

sudowind
authored and
dg845
committed
f0c0f00
Support ControlNet v1.1 shuffle properly (#3340)

authored and
dg845
committed
42eabb8
[Tests] better determinism (#3374)

sayakpaul
authored and
dg845
committed
1965acf
[docs] Add transformers to install (#3388)

stevhliu
authored and
dg845
committed
7b7b6bf
[deepspeed] partial ZeRO-3 support (#3076)

authored and
dg845
committed
c998614
Add omegaconf for tests (#3400)

patrickvonplaten
authored and
dg845
committed
1085f3e
Fix various bugs with LoRA Dreambooth and Dreambooth script (#3353)

authored and
dg845
committed
188de89
Fix docker file (#3402)

patrickvonplaten
authored and
dg845
committed
89a8f73
fix: deepseepd_plugin retrieval from accelerate state (#3410)

sayakpaul
authored and
dg845
committed
cb4016d
[Docs] Add `sigmoid` beta_scheduler to docstrings of relevant Schedulers (#3399)

authored and
dg845
committed
41763f6
Don't install accelerate and transformers from source (#3415)

patrickvonplaten
authored and
dg845
committed
622c3c6
Don't install transformers and accelerate from source (#3414)

patrickvonplaten
authored and
dg845
committed
f001e07
Improve fast tests (#3416)

patrickvonplaten
authored and
dg845
committed
80c2e55
attention refactor: the trilogy (#3387)

williamberman
authored and
dg845
committed
d749d57
[Docs] update the PT 2.0 optimization doc with latest findings (#3370)

authored and
dg845
committed
6ce7f8f
Fix style rendering (#3433)

pcuenca
authored and
dg845
committed
480b525
unCLIP scheduler do not use note (#3417)

williamberman
authored and
dg845
committed
d3b3855
Replace deprecated command with environment file (#3409)

authored and
dg845
committed
9a31cce
fix warning message pipeline loading (#3446)

patrickvonplaten
authored and
dg845
committed
df625f4
add stable diffusion tensorrt img2img pipeline (#3419)

asfiyab-nvidia
authored and
dg845
committed
8065462
Refactor controlnet and add img2img and inpaint (#3386)

patrickvonplaten
authored and
dg845
committed
d5f65dc
[Scheduler] DPM-Solver (++) Inverse Scheduler (#3335)

authored and
dg845
committed
2b11926
[Docs] Fix incomplete docstring for resnet.py (#3438)

Laurent2916
authored and
dg845
committed
63abfce
fix tiled vae blend extent range (#3384)

superlabs-dev
authored and
dg845
committed
32162aa
Small update to "Next steps" section (#3443)

pcuenca
authored and
dg845
committed
3f5a176
Allow arbitrary aspect ratio in IFSuperResolutionPipeline (#3298)

authored and
dg845
committed
3019e08
Adding 'strength' parameter to StableDiffusionInpaintingPipeline (#3424)

authored and
dg845
committed
bb1172b
[WIP] Bugfix - Pipeline.from_pretrained is broken when the pipeline is partially downloaded (#3448)

vimarshc
authored and
dg845
committed
2a16062
Fix gradient checkpointing bugs in freezing part of models (requires_grad=False) (#3404)

authored and
dg845
committed
68a97bd
Make dreambooth lora more robust to orig unet (#3462)

patrickvonplaten
authored and
dg845
committed
ce072e0
Reduce peak VRAM by releasing large attention tensors (as soon as they're unnecessary) (#3463)

cmdr2
authored and
dg845
committed
ee10c71
Add min snr to text2img lora training script (#3459)

wfng92
authored and
dg845
committed
9388b3a
Add inpaint lora scale support (#3460)

Glaceon-Hyy
authored and
dg845
committed
2ef1b00
[From ckpt] Fix from_ckpt (#3466)

patrickvonplaten
authored and
dg845
committed
62d9c72
Update full dreambooth script to work with IF (#3425)

williamberman
authored and
dg845
committed
368f9ad
Add IF dreambooth docs (#3470)

williamberman
authored and
dg845
committed
68441bf
parameterize pass single args through tuple (#3477)

williamberman
authored and
dg845
committed
eb7ae28
attend and excite tests disable determinism on the class level (#3478)

williamberman
authored and
dg845
committed
bb1e25a
dreambooth docs torch.compile note (#3471)

authored and
dg845
committed
9a195d7
add: if entry in the dreambooth training docs. (#3472)

sayakpaul
authored and
dg845
committed
09ddb88
[docs] Textual inversion inference (#3473)

authored and
dg845
committed
e36596c
[docs] Distributed inference (#3376)

stevhliu
authored and
dg845
committed
147da83
[{Up,Down}sample1d] explicit view kernel size as number elements in flattened indices (#3479)

williamberman
authored and
dg845
committed
2d8e089
mps & onnx tests rework (#3449)

pcuenca
authored and
dg845
committed
53e37b8
[Attention processor] Better warning message when shifting to `AttnProcessor2_0` (#3457)

sayakpaul
authored and
dg845
committed
8eae86d
[Docs] add note on local directory path. (#3397)

authored and
dg845
committed
55ca69b
Refactor full determinism (#3485)

patrickvonplaten
authored and
dg845
committed
a8219e8
Fix DPM single (#3413)

authored and
dg845
committed
a3e1153
Add `use_Karras_sigmas` to DPMSolverSinglestepScheduler (#3476)

Isotr0py
authored and
dg845
committed
01b42e4
Adds local_files_only bool to prevent forced online connection (#3486)

w4ffl35
authored and
dg845
committed
d22535a
make style

patrickvonplaten
authored and
dg845
committed
b78e854
[Docs] Korean translation (optimization, training) (#3488)

authored and
dg845
committed
a9ac5a8
DataLoader respecting EXIF data in Training Images (#3465)

Ambrosiussen
authored and
dg845
committed
634cf1f
make style

patrickvonplaten
authored and
dg845
committed
5782887
feat: allow disk offload for diffuser models (#3285)

authored and
dg845
committed
f61028f
[Community] reference only control (#3435)

okotaku
authored and
dg845
committed
30329a2
Support for cross-attention bias / mask (#2634)

Birch-san
authored and
dg845
committed
ca87f4d
do not scale the initial global step by gradient accumulation steps when loading from checkpoint (#3506)

williamberman
authored and
dg845
committed
cdf38f1
Remove CPU latents logic for UniDiffuserPipelineFastTests.
dg845
committed
51f0951
make style
dg845
committed
4ccb2b5
Revert "Clean up code and make slow tests pass."
dg845
committed
97e8eef
Batched load of textual inversions (#3277)

authored and
dg845
committed
9f84416

Commits on May 22, 2023

Revert bad commit and clean up code.
dg845
committed
9f7247c

Commits on May 23, 2023

Commits on May 24, 2023

Commits on May 25, 2023

Commits on May 26, 2023