Image and video understanding adapted to your own specialized tasks
NOTE
These instructions target Linux with a GPU-enabled Docker container. Run training and Streamlit steps inside the container unless noted otherwise. The same container supports both the image and video recipes.
To manage containers without sudo, your user must be in the docker group. If you skip this step, run Docker commands with sudo.
Open a terminal and test Docker access:
docker ps
If you see a permission-denied error (for example, permission denied while trying to connect to the Docker daemon socket), add your user to the docker group:
sudo usermod -aG docker $USER
newgrp docker
Clone the playbook assets and open the assets directory:
git clone https://github.com/NVIDIA/dgx-spark-playbooks
cd dgx-spark-playbooks/nvidia/playbook-vlm-finetuning/assets
Export your Hugging Face token, then build the image. Warnings during the build are expected and can be ignored.
export HF_TOKEN=<YOUR_HF_TOKEN>
docker build --build-arg HF_TOKEN=$HF_TOKEN -t vlm_demo .
sh launch.sh
cd /vlm_finetuning
NOTE
The same Docker container and launch commands work for both image and video VLM recipes. The image includes dependencies such as FFmpeg, Decord, and libraries used by both workflows.
Choose Option A (image) or Option B (video) below. You can run both over time in the same container, but stop Streamlit or Jupyter and free memory before switching heavy workloads.
hf download Qwen/Qwen2.5-VL-7B-Instruct
If you already have a fine-tuned checkpoint, place it in the saved_model/ folder. Checkpoint numbers can differ. For a comparative analysis against the base model only, skip ahead to Fine-tuned model inference.
This recipe uses a wildfire prediction dataset with satellite and aerial imagery for binary classification (wildfire vs no wildfire).
mkdir -p ui_image/data
cd ui_image/data
Open the Wildfire Prediction Dataset on Kaggle. Use Download Via → cURL, copy the curl command, then run it in the container.
NOTE
You must be logged into Kaggle and may need to accept the dataset terms before the download link works.
# Paste and run the curl command from Kaggle, then unzip
unzip -qq wildfire-prediction-dataset.zip
rm wildfire-prediction-dataset.zip
cd ..
Start the Streamlit demo to evaluate the base model before fine-tuning:
streamlit run Image_VLM.py
Open http://localhost:8501/ (or http://<HARDWARE_IP>:8501 from another device).
On first access, the backend starts vLLM servers for the base model. A spinner appears while vLLM comes up; this can take up to about 15 minutes.
Scroll to the Image Inference section. A sample satellite image is pre-loaded. Enter a prompt and select Generate. The left panel shows the base model; the right panel stays empty until a fine-tuned model is available. Example prompt:
Identify if this region has been affected by a wildfire
The base model typically struggles on this domain-specific task. Use GRPO fine-tuning next to improve accuracy and add structured reasoning.
With the Streamlit demo running, scroll to the GRPO Training section.
Configure fine-tuning method and LoRA parameters:
Enable the VLM layers you want to train. For best quality, leave all options on (this increases training time).
Training parameters:
GRPO reward settings:
Select Start Finetuning. Allow about 15 minutes for the model to load and for metadata to appear in the UI. Loss, step, and GRPO rewards update in a live table as training runs.
The default configuration is a reasonable starting point: about 100 steps can take up to about 2 hours. Longer runs (for example around 1000 steps) can take much longer and may improve accuracy further.
When training finishes, the script merges LoRA weights into the base model. Merging can take about 5 minutes after the last step.
To interrupt training, use Stop Finetuning. Use that control only to interrupt a run; it does not guarantee that checkpoints are stored or that LoRA adapters are fully merged.
After you stop training, the UI brings up vLLM servers for the base model and the newly fine-tuned model.
Compare the base and fine-tuned models. If Streamlit is not already running:
streamlit run Image_VLM.py
Wait about 15 minutes for the vLLM servers to come up if they are not already loaded.
In Image Inference, enter a prompt and select Generate. Example:
Identify if this region has been affected by a wildfire
With sufficient training, the fine-tuned model should reason in markdown and end with a concise bolded answer.
Inside the same container, open the video UI directory:
cd /vlm_finetuning/ui_video
Structure the dataset as follows. metadata.jsonl must contain one structured JSON record per video:
dataset/
├── videos/
│ ├── video1.mp4
│ ├── video2.mp4
│ └── ...
└── metadata.jsonl
hf download OpenGVLab/InternVL3-8B
Evaluate base InternVL3-8B before fine-tuning:
# cd into /vlm_finetuning/ui_video if you are not already there
streamlit run Video_VLM.py
Open http://localhost:8501/ (or http://<HARDWARE_IP>:8501).
On first access, the backend loads the base model through Hugging Face. A spinner appears; loading can take up to about 10 minutes.
Select a video from the dashcam gallery (green open icon). The video plays for reference. Enter a prompt and select Generate. The left panel shows the base model; the right panel stays empty until a fine-tuned model is available. Example prompt:
Analyze the dashcam footage for unsafe driver behavior
Before training, stop the Streamlit demo with Ctrl+C in the terminal.
NOTE
To clear buffer cache after stopping the Streamlit UI (outside the container):
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
cd train
jupyter notebook video_vlm.ipynb
Open Jupyter at http://localhost:8888 (or http://<HARDWARE_IP>:8888). Set the dataset path in the notebook:
dataset_path = "/path/to/your/dataset"
Key training knobs:
Usable video quality often needs a long training window (on the order of a day or more) because of spatio-temporal sequence processing. Monitor metrics in the notebook as training runs.
When finished, shut down the Jupyter kernel and stop the Jupyter server with Ctrl+C.
NOTE
To clear buffer cache after stopping Jupyter (outside the container):
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
Compare base and fine-tuned models. If Streamlit is not already running:
# cd back to /vlm_finetuning/ui_video if needed
streamlit run Video_VLM.py
Open http://localhost:8501/. With sufficient training, the fine-tuned model should surface salient events and emit structured output that matches the schema you trained—suitable for export into a video analytics database. Try additional gallery videos as needed.
Stop running Streamlit or Jupyter with Ctrl+C. Exit the container, then remove the local image if you no longer need it:
docker rmi vlm_demo
WARNING
Removing the image deletes the local Docker build. Rebuild with the Dockerfile in assets/ before running this playbook again. Downloaded models, datasets, and checkpoints under the mounted assets directory are separate; delete those only if you intend to free disk space.