VisionCaptioner

VisionCaptioner is a local desktop application designed to automate the creation of detailed captions for image & video datasets.

Built specifically for AI researchers and enthusiasts training custom models (LoRA, Fine-tuning, Flux, Z-Image Turbo, Qwen-Image, SDXL, Wan, HunyuanVideo etc). VisionCaptioner leverages Vision-Language Models from the Qwen-VL and Google Gemma 4 families to generate high-quality, context-aware descriptions in batch.

✨ Features

• User-Friendly Interface: Simple GUI to manage your image & video captioning tasks.
• Caption generation Automatically generate captions using Qwen-VL or Google Gemma 4 models.
  • Video Support: Unlike other tools, this analyzes video files by extracting multiple frames to understand motion and context.
  • LoRA Friendly: Includes features specifically for training, such as Trigger Word injection and skipping existing caption files.
  • Review & Edit Quickly review and manually edit your captions on a dataset.
  • Find & Replace Functionality to find and replace strings in your captions dataset, including often used presets.
  • System Prompts: Choose from built-in presets (tuned for various models) or write your own custom instructions.
  • Resolution & Quantization: Adjustable settings to balance between speed, VRAM usage, and descriptive detail.
  • Works with many Vision-Language models Qwen2.5-VL, Qwen3-VL, Google Gemma 4 (E2B/E4B/26B-A4B/31B), base models, Abliterated versions, GGUF models (Qwen only for now)
• Masking Support: Functionality to create mask files using Segment Anything 3 model with promptable subject.
  • Editing Functionality to quickly paint/edit/extract/contract masks in a visual editor.
  • Different mask formats Masks can be saved as separate files or embedded in the image files.
  • Compatibility with OneTrainer
  • Potentially compatible with other/future training tools
• Video Extraction Functionality to extract frames from videos containing a prompted subject.
• Quality Assurance Analyze images for blur, resolution, face/eyes detection, and missing masks.
  • Scoring Weighted quality scoring with sortable table view.
  • Batch Actions Apply actions to all images below a quality threshold.
• Local Execution: Runs entirely on your machine for privacy and control.
• CommandLine Interface: Option to use this from the commandline and/or scripts.

🛠️ Installation

Prerequisites

• Python 3.10+
• NVIDIA GPU (with CUDA).
• AMD GPU (with ROCm on Linux)
• Apple Silicon Mac (M-series, with Metal / MPS)

Windows Setup

git clone https://github.com/Brekel/VisionCaptioner.git
cd VisionCaptioner
python -m venv venv
.\venv\Scripts\activate
python -m pip install --upgrade pip
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130
pip install -r requirements.txt

AMD GPU with ROCm support for Windows, more info here: https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/windows/install-pytorch.html

Linux Setup

Python Setup:

git clone https://github.com/Brekel/VisionCaptioner.git
cd VisionCaptioner
python3 -m venv venv
source venv/bin/activate
python3 -m pip install --upgrade pip
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu130
pip3 install -r requirements.txt

AMD GPU with ROCm support should work under Linux when using different pytorch install command:

pip3 install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1

Qt/GUI Dependencies (Ubuntu/Debian):

Some Linux systems may require additional packages for the Qt-based GUI to work:

sudo apt install libxcb-cursor0 libxcb-xinerama0 libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-shape0 libxcb-xfixes0 libxcb-xkb1 libxkbcommon-x11-0

macOS Setup (Apple Silicon)

Runs on M-series Macs using Metal / MPS. Install the default PyTorch wheels (they ship MPS support, so there is no CUDA index URL):

git clone https://github.com/Brekel/VisionCaptioner.git
cd VisionCaptioner
python3 -m venv venv
source venv/bin/activate
python3 -m pip install --upgrade pip
pip install torch torchvision
pip install -r requirements.txt

Notes:

• NVIDIA/CUDA-only packages (bitsandbytes, nvidia-ml-py, triton) are skipped automatically on macOS, so the Int8/NF4 quantization options are unavailable (the app loads full precision instead). Use None (BF16) or FP16, or a GGUF model.
• The FP8 quantization option is off by default: it needs an NVIDIA GPU with compute capability 8.9+ (RTX 4090 or newer) and the optional kernels package (pip install "kernels<0.15" - 0.15+ breaks import transformers). Without either, FP8 is greyed out and the tooltip says why. On Windows it is slower than both None (BF16) and NF4, because the fast FP8 kernel (DeepGEMM) has no Windows build; prefer NF4 for VRAM savings.
• Transformers models run on MPS. For GGUF models, use the built-in installer on the Captions tab (📥 button): it fetches a prebuilt Metal llama-cpp-python wheel (pip install llama-cpp-python does not work — see readme_models.md).
• Launch with python main.py or ./run.sh.

Troubleshooting: inconsistent Name error while installing PyTorch

If pip install torch torchvision --index-url ... fails with:

Requested typing-extensions>=4.10.0 ... has inconsistent Name:
expected 'typing-extensions', but metadata has 'typing_extensions'
...
ERROR: Cannot install torch==... because these package versions have conflicting dependencies.

your pip is too old: it compares the raw package name instead of normalizing it, and typing_extensions / typing-extensions are the same package. A fresh venv seeds whatever pip shipped with your Python (Python 3.10 ships pip 23.0.1), which is why this hits on a clean install. Upgrading pip inside the venv is the fix, so do not skip this line:

python -m pip install --upgrade pip

If you cannot upgrade pip, pre-install the affected packages from PyPI before the torch command:

pip install typing-extensions jinja2

Do not work around it by switching --index-url to --extra-index-url. pip does not prioritize one index over the other — it merges both and picks the highest version, so you will silently get the CPU-only torch from PyPI instead of the CUDA/ROCm build.

Update

The easiest way to update is to use the provided update scripts, which will pull the latest code from git and upgrade your Python packages in one step:

• Windows: double-click update.bat (or run it from a terminal)
• Linux / macOS: ./update.sh (you may need to chmod +x update.sh once)

Or update manually:

cd VisionCaptioner
git pull
venv\Scripts\activate          # Windows
# source venv/bin/activate     # Linux / macOS
pip install --upgrade -r requirements.txt

🚀 Launching the Application

You can start the application using the provided scripts or manually via Python.

Start manually:

python main.py

Or using the run.bat (Windows) or run.sh (Linux) scripts

📥 Download/Install Models

• Models can be installed using the built-in downloads manager on the Captions tab (📥💾 button).
• Alternatively, you can manually download models from HuggingFace into the /models folder.
• More info in the readme_models.md file.

📖 General Usage

• First select your Image/Video folder at the top (using Browse button or drag & drop)
• Note that all settings have a tooltip description if you hover your mouse over them
• Use the Captions tab to generate captions for images and videos
• Use the Review & Edit tab to review and edit captions
• Use the Video Extraction tab to extract frames from videos
• Use the Mask Segmentation tab to create masks for images (this is optional)
• Use the Quality Assurance tab to analyze and curate image quality in your dataset

🖥️ CLI - Command Line Interface usage

• For advanced users, the tool can be used from the commandline or scripts
• It will automatically read the settings.json file generated by the user
• Parameters can be overruled on the commandline
• Please refer to commandline_interface.md for documentation

Acknowledgements

Qwen Team: Alibaba Cloud - For developing and open-sourcing the powerful Qwen-VL models.

Google DeepMind - For developing and open-sourcing the Gemma 4 family of multimodal models.

Meta AI - For developing and open-sourcing the Segment Anything Model 3 (SAM3).

This project was inspired by:

AI Lab's ComfyUI-QwenVL node for ComfyUI:

• https://github.com/1038lab/ComfyUI-QwenVL

OneTrainer and its masked training features:

• https://github.com/Nerogar/OneTrainer

---

🙏 Citation & Support

If you find this tool useful in your research or projects, please consider:

• Giving a ⭐ on GitHub.
• Citing the project as: Brekel - VisionCaptioner (https://brekel.com).
• Follow on Twitter/X: https://x.com/brekelj
• Checking out other tools at brekel.com.