Repository navigation
train_dreambooth_lora.py -- ValueError: Attempting to unscale FP16 gradients caused by "--validation_prompt" param. #13124
Description
Describe the bug
Hello, when I directly run the training script provided in examples/dreambooth, with the following command, I encounter the “Attempting to unscale FP16 gradients” error:
`accelerate launch examples/dreambooth/train_dreambooth_lora.py \
--mixed_precision="fp16" \
--pretrained_model_name_or_path=$MODEL_NAME \
--instance_data_dir=$INSTANCE_DIR \
--output_dir=$OUTPUT_DIR \
--instance_prompt="a photo of sks dog" \
--resolution=512 \
--train_batch_size=1 \
--gradient_accumulation_steps=1 \
--checkpointing_steps=100 \
--learning_rate=1e-4 \
--report_to="tensorboard" \
--lr_scheduler="constant" \
--lr_warmup_steps=0 \
--max_train_steps=500 \
--seed="0" \
--validation_prompt="A photo of sks dog in a bucket" \
--validation_epochs=50`
I then tried the solution proposed in PR #6554 and found that the following command does not trigger this issue:
> > `accelerate launch examples/dreambooth/train_dreambooth_lora.py \
> > --mixed_precision="fp16" \
> > --pretrained_model_name_or_path=$MODEL_NAME \
> > --instance_data_dir=$INSTANCE_DIR \
> > --output_dir=$OUTPUT_DIR \
> > --instance_prompt="a photo of sks dog" \
> > --resolution=512 \
> > --train_batch_size=1 \
> > --gradient_accumulation_steps=1 \
> > --checkpointing_steps=100 \
> > --learning_rate=1e-4 \
> > --lr_scheduler="constant" \
> > --lr_warmup_steps=0 \
> > --max_train_steps=500 \
> > --gradient_checkpointing \
> > --seed="0" \
> > --report_to="tensorboard"`
>
I then tried to align the two commands and noticed that the issue is triggered specifically when the --validation_prompt argument is provided.
Does this indicate that there is still a bug related to validation in this script?
Reproduction
`accelerate launch examples/dreambooth/train_dreambooth_lora.py \
--mixed_precision="fp16" \
--pretrained_model_name_or_path=$MODEL_NAME \
--instance_data_dir=$INSTANCE_DIR \
--output_dir=$OUTPUT_DIR \
--instance_prompt="a photo of sks dog" \
--resolution=512 \
--train_batch_size=1 \
--gradient_accumulation_steps=1 \
--checkpointing_steps=100 \
--learning_rate=1e-4 \
--report_to="tensorboard" \
--lr_scheduler="constant" \
--lr_warmup_steps=0 \
--max_train_steps=500 \
--seed="0" \
--validation_prompt="A photo of sks dog in a bucket" \
--validation_epochs=50`
Logs
System Info
- 🤗 Diffusers version: 0.37.0.dev0
- Platform: Linux-5.4.0-216-generic-x86_64-with-glibc2.35
- Running on Google Colab?: No
- Python version: 3.10.12
- PyTorch version (GPU?): 2.9.1+cu128 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Huggingface_hub version: 1.4.1
- Transformers version: 5.1.0
- Accelerate version: 1.12.0
- PEFT version: 0.18.1
- Bitsandbytes version: not installed
- Safetensors version: 0.7.0
- xFormers version: not installed
- Accelerator: NVIDIA GeForce RTX 3060, 12288 MiB
- Using GPU in script?:
- Using distributed or parallel set-up in script?:
Who can help?
No response
Activity
zhanglin2603 commentedon Feb 17, 2026
Hey, I went over #13124 and can take this on.
If this approach works for you, I can start now.
zhanglin2603 commentedon Feb 18, 2026
Quick follow-up on #13124: I am still available to work on this issue.
I can provide an initial implementation update within 24 hours and keep the scope tight.
If assignment changed, please let me know and I will avoid adding noise.
Fix `train_dreambooth_lora.py` fp16 unscale error after validation
Fix fp16 LoRA unscale crash after validation in train_dreambooth_lora.py
Fix fp16 LoRA unscale crash after validation in train_dreambooth_lora…
Fix fp16 LoRA unscale crash after validation in train_dreambooth_lora…