---
title: "Fine-Tune Vision Language Models — Overview"
canonical: "https://build.nvidia.com/playbooks/vlm-finetuning/overview.md"
---

# Basic idea

This playbook shows how to fine-tune Vision-Language Models (VLMs) for image and video understanding on your **hardware platform**. You adapt multimodal models to specialized visual tasks with LoRA, preference optimization, and interactive Streamlit UIs for training and side-by-side inference.

Two recipes are included:

- **Image VLM fine-tuning** — Qwen2.5-VL-7B for wildfire detection from satellite imagery, using GRPO (Generalized Reward Preference Optimization)
- **Video VLM fine-tuning** — InternVL3-8B for dangerous driving analysis and structured metadata generation from driving videos

Both paths use Docker for a reproducible environment with the libraries needed for image and video workflows.

# What you'll accomplish

You'll have fine-tuned VLM checkpoints for image and/or video tasks on your **hardware platform**, with interactive UIs for training and comparing base vs fine-tuned models.

- Fine-tune Qwen2.5-VL for wildfire detection with GRPO and LoRA
- Fine-tune InternVL3 for dangerous driving analysis and structured video metadata
- Run Streamlit demos for base and fine-tuned inference side by side
- Use a shared Docker image for both image and video recipes

# What to know before starting

**Required:**

- Basic Linux command line and Docker container usage
- Familiarity with Hugging Face model downloads and tokens
- Fine-tuning concepts: LoRA, checkpoints, and GPU memory constraints

**Optional:**

- Experience with vision-language models and multimodal prompts
- Dataset prep for image classification or video + JSONL metadata
- Weights & Biases for training monitoring

# Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

| Hardware platform | OS | Memory | Recommended default local settings | Multi-node capable hardware |
| :---- | :---- | :---- | :---- | :---- |
| **DGX Spark** | DGX OS (Linux) | 128 GB Unified Memory | Docker image `vlm_demo` built from playbook assets | — |

# Prerequisites

**Hardware requirements**

- Supported hardware platform — see Supported hardware platforms matrix above
- Sufficient memory for 7B–8B VLM training and inference with vision encoders loaded
- Enough free disk for model downloads and datasets (plan for tens of GB)
- No other heavy GPU workloads running during training or inference

**Software requirements**

- NVIDIA Docker / NVIDIA Container Toolkit: `docker --version` and `nvidia-smi` inside a GPU container
- Network access to download models from Hugging Face and datasets (for example Kaggle)
- Hugging Face access token (`HF_TOKEN`) for model downloads
- Web browser access to ports `8501` (Streamlit) and `8888` (Jupyter, video recipe)
- Weights & Biases account for training monitoring (optional but recommended)

# Ancillary files

All required assets can be found [in this playbook repository](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/playbook-vlm-finetuning/).

- `assets/Dockerfile` / `assets/launch.sh` — Build and run the shared VLM fine-tuning container
- `assets/ui_image/` — Streamlit app, configs, and scripts for image VLM (Qwen2.5-VL + GRPO)
- `assets/ui_video/` — Streamlit app, training notebook, and configs for video VLM (InternVL3)
- `assets/README.md` — Asset-level overview of both recipes

# Time & risk

- **Estimated time:** 2 HOURS for setup plus a short image training run with the default UI settings; video fine-tuning for usable quality typically needs a much longer run (often a day or more depending on dataset and hyperparameters)
- **Risk level:** Medium
- Docker permission issues may require a group change and new login session
- Large model and dataset downloads need significant disk space and time
- Training sustains high GPU and memory use; residual cache can cause pressure after UI or notebook sessions
- Image dataset prep may require a Kaggle login and accepting dataset terms; video datasets need correct folder layout and `metadata.jsonl`
- **Rollback:** Stop and remove Docker containers; delete downloaded models, datasets, and checkpoints under the mounted assets directory if needed
- **Last Updated:** 10/09/2026
- Pinned `peft` in the Dockerfile so GRPO fine-tuning no longer fails on the base image's `torchao` version, and added a troubleshooting entry