When Preference Labels Fall Short:
Aligning Diffusion Models from Real Data

📝 Abstract

Preference alignment aims to guide generative models by learning from comparisons between preferred and non-preferred samples. In practice, most existing approaches rely on preference pairs constructed from model-generated images. Such supervision is inherently relative and can be ambiguous when both samples exhibit artifacts or limited visual quality, making it difficult to infer what constitutes a truly desirable output. In this work, we investigate whether real data can serve as an alternative source of supervision for preference alignment. We adopt a data-centric perspective and study a curation strategy that treats real images as reference points and constructs preference signals by contrasting them with generated or perturbed samples, without requiring manually annotated preference pairs. Through empirical analysis, we show that real-data-based supervision provides effective guidance for aligning diffusion models and achieves performance comparable to existing preference-based methods. Our results suggest that real data offers a practical and complementary source of supervision for preference alignment and highlight directions of label-efficient alignment strategies.

🖼️ Gallery

🗂️ Repository layout

Only the directories below are part of RealAlign itself. Each top-level folder is an independent subproject with its own scripts and dependencies.

Data curation

• data_curation/ builds (real, fake) preference pairs from HPDv3, Pick-a-Pic v2, and Civitai-top.
• The pipeline follows four stages: extract/ → construct_pairs/ → score/ → filter/.
• The curated CSV is consumed by both SD-1.5 and SD-3.5-M training.

Training

• training_sd15/ contains the RealAlign SD-1.5 two-stage trainers:
  • stage1_diffusion_dro/train-irl.py for Stage 1 Diffusion-DRO / inverse RL.
  • stage2_dpo/train-lora_init.py for Stage 2 Diffusion-DPO, warm-started from the Stage 1 LoRA.
• training_sd35m/scripts/ contains the RealAlign SD-3.5-M two-stage trainers:
  • train-sd-3-5-medium-irl.py for Stage 1.
  • train-sd-3-5-medium-dpo.py for Stage 2.

Evaluation

• training_sd35m/evaluation/ contains the reward-model evaluation harness for both SD-1.5 and SD-3.5-M.
  • training_sd35m/evaluation/sd-v1-5/ provides SD-1.5 generate_image.py, calculate_score.py, and run_multi_seed_eval.sh.
  • training_sd35m/evaluation/sd-3-5-medium/ provides SD-3.5-M generate_image.py, calculate_score.py, and run_multi_seed_eval.sh.
  • Prompt lists live under training_sd35m/dataset/, including pick_a_pic_v2/, partiprompts/, and drawbench-unique/.
  • All six reward metrics, PickScore, ImageReward, Aesthetic, HPSv3, DeQA, and UnifiedReward, are routed through flow_grpo.rewards.multi_score.
• DPG-Bench/ contains DPG-Bench evaluation scripts for SD-1.5 and SD-3.5-M.

Paper artifacts

• notebook/ contains curated ICML 2026 analysis notebooks and figure artifacts used in the paper.

🚀 Quick start

1. Environment Set Up

Create and activate the shared alignprop environment:

conda create -n alignprop python=3.10
conda activate alignprop
pip install -r requirements.txt

2. Build training pairs

Run the four-stage pipeline in data_curation/ (extract → construct_pairs → score → filter) to produce the paired (real, fake) training CSV. Full details: data_curation/README.md.

3. Train

SD-1.5

# Stage 1: Diffusion-DRO
bash training_sd15/stage1_diffusion_dro/train-irl.sh
# Stage 2: Diffusion-DPO (LoRA-init)
bash training_sd15/stage2_dpo/lora_init.sh

SD-3.5-M

# Stage 1: Diffusion-DRO
bash training_sd35m/scripts/single_node/inverse_reinforcement_learning.sh
# Stage 2: Diffusion-DPO (LoRA-init)
bash training_sd35m/scripts/single_node/dpo.sh

• Shared input. Both stages read the same CSV — csv_file_path_train for SD-1.5, config.{irl,dpo}.csv_file_path for SD-3.5-M. SD-3.5-M also expects precomputed prompt embeddings (see training_sd35m/README.md).
• Stage 2 warm-starts from Stage 1. Before launching Stage 2, point it at the Stage 1 LoRA checkpoint: set pretrained_lora_path in training_sd15/stage2_dpo/lora_init.sh (SD-1.5) or config.train.lora_path in training_sd35m/config/sd3_5_medium_dpo.py (SD-3.5-M).

Full hyperparameters, launchers, and config schema: training_sd15/README.md, training_sd35m/README.md.

4. Evaluate

Reward-model evaluation lives in training_sd35m/evaluation/ for both SD-1.5 and SD-3.5-M.

DPG-Bench generation + evaluation scripts live in DPG-Bench/ (DPG-Bench-script-sd-v1-5.sh, DPG-Bench-script-sd-3-5-medium.sh).

🙏 Acknowledgement

Our codebase references the code from Diffusion-DRO, Diffusion-DPO, and Flow-GRPO.

We thank the authors for releasing their implementations.

📚 Citation

If you find our work useful, please consider citing:

@article{chen2026preference,
  title={When Preference Labels Fall Short: Aligning Diffusion Models from Real Data},
  author={Chen, Weiyan and Deng, Weijian and Xiao, Yao and Tu, Weijie and Dong, ZiYi and Radwan, Ibrahim and Lin, Liang and Wei, Pengxu},
  journal={arXiv preprint arXiv:2605.19839},
  year={2026}
}