MedSynV2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts

Weicheng Dai¹, Chenyu Wang¹, Shantanu Ghosh¹, Kayhan Batmanghelich^1*
1Department of Electrical and Computer Engineering, Boston University
^*Corresponding author

ECCV 2026

📄 Read the manuscript (ECCV 2026)

---

🧠 Overview

MedSynV2 is a flexible multimodal framework for controllable 3D medical image generation, designed to overcome the limitations of existing text-only or segmentation-only conditioning methods.

Our model supports optional and partial conditioning from:

• 📝 Radiology reports (semantic, flexible)
• 🧩 Segmentation prompts (precise, spatial)

Crucially, MedSynV2 does not require full-organ annotations.
Users may provide segmentation for a specific anatomy or abnormality, whose semantic meaning is specified through an accompanying text description.

This design enables scalable, fine-grained control over volumetric generation while maintaining high image fidelity. For the full technical details, method description, and experimental results, please see our manuscript.

---

✨ Key Features

• 🔀 Multimodal conditioning: text-only, segmentation-only, or both
• 🧩 Partial segmentation support (no full-organ masks required)
• 🧠 Strong semantic grounding via text-described segmentation prompts
• ⚡ Memory-efficient diffusion transformer for high-resolution 3D volumes
• 🔍 Gated attention for long radiology reports
• 🧬 Generates anatomically consistent, high-resolution CT volumes

---

🧩 Method Overview

📄 View full-resolution PDF

MedSynV2 extends diffusion transformers to jointly process:

• Image tokens
• Segmentation tokens
• Text tokens from radiology reports

A gated attention mechanism enables effective conditioning on long, unstructured clinical text, while preserving spatial controllability through segmentation prompts.

---

📊 Results Summary

We evaluate MedSynV2 on large-scale 3D CT datasets, using:

• Perceptual metrics (FID, SSIM, PSNR)
• Semantic consistency metrics
• Radiologist evaluation

Key findings:

• 🚀 ~24% relative improvement in mean FID
• 🧠 Strong semantic alignment between generated and real CT volumes
• 🧬 High-resolution, anatomically coherent synthesis
• 📈 Improved data efficiency when used for data augmentation
• 🌱 Generalization to concepts beyond training distribution

---

📦 Code Status

The full release will include:

• Model architecture and diffusion transformer implementation
• Multimodal conditioning pipelines
• Inference scripts

0. Create conda environment

conda env create -f environment.yml 

If flash-attention raises issue, please follow the original github and compile flash-attn==2.7.4.post1.

1. Download pretrained models

pretrained model components	Checkpoints
Text model (same as MedSyn)	pretrained_lm
MedSynV2 Generator	final_ckpt
OpenSora checkpoint	vae.ckpt

2.1 If you prefer report, extract text features

Here's an example.

cd src
python extract_text_feature.py --prompt 'There is no airspace opacity, effusion or pneumothorax. There is no evidence of suspicious pulmonary nodule or mass.' \
                               --text_model_path './model/pretrained_lm' \
                               --save_path './result/text_feature/normal.npy'

2.2 Latent feature generation

cd src
python medsynv2_DiT_reportonly.py --text_feature_folder './text_feature' \
                                  --pretrain_model_path ./model/medsynv2_dit_x0.pth \
                                  --save_path './tmp/medsynv2_dit_x0_results_reportonly'

2.3 OpenSora decoding

cd opensora/osp
python examples/recon_ct.py --pretrained_model_path './model/vae.ckpt' \
                            --folder './tmp/medsynv2_dit_x0_results_reportonly' \
                            --save_folder './tmp/medsynv2_dit_x0_results_reportonly_SR' 

if you need to convert it to nifti

python examples/convert.py --folder './tmp/medsynv2_dit_x0_results_reportonly_SR' \
                           --save_folder './tmp/medsynv2_dit_x0_results_reportonly_SR_nii' 

3.1 If you need masks encoded, encode the mask first

We encode the masks with batch_index from 0 to 5, as a 48GB GPU will only handle this size. After running, concatenate them along the depth axis (trivial).
Make sure your masks are binary masks.
Note we extract contours of lobes with encode_masks_lobe.py

cd opensora/osp
python examples/encode_masks.py --batch_index 0 \ # you should use 0~5 here
                                --pretrained_model_path './model/vae.ckpt' \
                                --folder 'YOUR NIFTI MASKS' \
                                --save_folder 'YOUR MASKS_LATENTS FOLDER' 

3.2 (Optional) If you need reports as well, follow 2.1 above

Otherwise create some dummy npy files in the text_feature folder, just as file name indicator.

3.3 Latent feature generation with mask features (and reports)

Make sure you check line 559 and put corresponding masks into each folders. The pre-conditions (indicating which masks) are already extracted and uploaded.

cd src
python medsynv2_DiT_reportonly.py --text_feature_folder './text_feature' \
                                  --mask_folder 'YOUR MASKS_LATENTS FOLDER' \
                                  --pretrain_model_path ./model/medsynv2_dit_x0.pth \
                                  --save_path './tmp/medsynv2_dit_x0_results_report_mask' \
                                  --include_report

anatomy / abnormality	corresponding suffix index
lobes	1
airways	2
vessel	3
heart	4
consolidation	5
ground glass opacity	6
pericardial effusion	7
pleural effusion	8
nodule	9

3.4 OpenSora Decoding

The same as 2.3

If you find any issue implementing the inference, please let us know

License and copyright

The source code is Licensed under Apache License 2.0. Model weights is Licensed under Custom Academic License for Model Weights.

---

📚 Citation

If you find this work useful, please cite:

@inproceedings{dai2026medsynv2,
  title     = {Flexible Multimodal Controllable Generation of 3D Medical Images},
  author    = {Dai, Weicheng and Wang, Chenyu and Ghosh, Shantanu and Batmanghelich, Kayhan},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Credits

The code is heavily based on MedSyn and OpenSoraPlan.