Fine-Tune Vision Language Models

Basic idea

This playbook shows how to fine-tune Vision-Language Models (VLMs) for image and video understanding on your hardware platform. You adapt multimodal models to specialized visual tasks with LoRA, preference optimization, and interactive Streamlit UIs for training and side-by-side inference.

Two recipes are included:

• Image VLM fine-tuning — Qwen2.5-VL-7B for wildfire detection from satellite imagery, using GRPO (Generalized Reward Preference Optimization)
• Video VLM fine-tuning — InternVL3-8B for dangerous driving analysis and structured metadata generation from driving videos

Both paths use Docker for a reproducible environment with the libraries needed for image and video workflows.

What you'll accomplish

You'll have fine-tuned VLM checkpoints for image and/or video tasks on your hardware platform, with interactive UIs for training and comparing base vs fine-tuned models.

• Fine-tune Qwen2.5-VL for wildfire detection with GRPO and LoRA
• Fine-tune InternVL3 for dangerous driving analysis and structured video metadata
• Run Streamlit demos for base and fine-tuned inference side by side
• Use a shared Docker image for both image and video recipes

What to know before starting

Required:

• Basic Linux command line and Docker container usage
• Familiarity with Hugging Face model downloads and tokens
• Fine-tuning concepts: LoRA, checkpoints, and GPU memory constraints

Optional:

• Experience with vision-language models and multimodal prompts
• Dataset prep for image classification or video + JSONL metadata
• Weights & Biases for training monitoring

Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

Prerequisites

Hardware requirements

• Supported hardware platform — see Supported hardware platforms matrix above
• Sufficient memory for 7B–8B VLM training and inference with vision encoders loaded
• Enough free disk for model downloads and datasets (plan for tens of GB)
• No other heavy GPU workloads running during training or inference

Software requirements

• NVIDIA Docker / NVIDIA Container Toolkit: docker --version and nvidia-smi inside a GPU container
• Network access to download models from Hugging Face and datasets (for example Kaggle)
• Hugging Face access token (HF_TOKEN) for model downloads
• Web browser access to ports 8501 (Streamlit) and 8888 (Jupyter, video recipe)
• Weights & Biases account for training monitoring (optional but recommended)

Ancillary files

All required assets can be found in this playbook repository.

• assets/Dockerfile / assets/launch.sh — Build and run the shared VLM fine-tuning container
• assets/ui_image/ — Streamlit app, configs, and scripts for image VLM (Qwen2.5-VL + GRPO)
• assets/ui_video/ — Streamlit app, training notebook, and configs for video VLM (InternVL3)
• assets/README.md — Asset-level overview of both recipes

Time & risk

• Estimated time: 2 HOURS for setup plus a short image training run with the default UI settings; video fine-tuning for usable quality typically needs a much longer run (often a day or more depending on dataset and hyperparameters)
• Risk level: Medium
  • Docker permission issues may require a group change and new login session
  • Large model and dataset downloads need significant disk space and time
  • Training sustains high GPU and memory use; residual cache can cause pressure after UI or notebook sessions
  • Image dataset prep may require a Kaggle login and accepting dataset terms; video datasets need correct folder layout and metadata.jsonl
• Rollback: Stop and remove Docker containers; delete downloaded models, datasets, and checkpoints under the mounted assets directory if needed
• Last Updated: 10/09/2026
  • Pinned peft in the Dockerfile so GRPO fine-tuning no longer fails on the base image's torchao version, and added a troubleshooting entry