Fine-Tune Vision Language Models

Common issues

The Hardware platform column shows where an issue is most relevant. "All hardware platforms" applies to every platform listed in this playbook.

Symptom	Hardware platform	Cause	Fix
"permission denied" when running Docker	All hardware platforms	User not in the `docker` group	Run `sudo usermod -aG docker $USER && newgrp docker`
Container fails to start with GPU error	All hardware platforms	NVIDIA Container Toolkit not configured	Configure the NVIDIA Container Toolkit for Docker and confirm `nvidia-smi` works in a GPU container
Hugging Face download fails or auth error	All hardware platforms	Missing or invalid `HF_TOKEN`	Export a valid Hugging Face token; pass `--build-arg HF_TOKEN=$HF_TOKEN` when building; run `hf auth login` inside the container if needed
Kaggle dataset curl fails	All hardware platforms	Not logged in or terms not accepted	Sign in to Kaggle, accept dataset terms, regenerate the cURL download command, and retry
Streamlit unreachable on port 8501	All hardware platforms	App not running or port blocked	Confirm `streamlit run` is active; open `http://localhost:8501` or `http://<HARDWARE_IP>:8501`
Jupyter unreachable on port 8888	All hardware platforms	Notebook server not running or port blocked	Confirm `jupyter notebook` is running in `ui_video/train`; open `http://localhost:8888` or `http://<HARDWARE_IP>:8888`
vLLM / model load spinner runs a long time	All hardware platforms	First-time model load or cold start	Wait up to about 15 minutes for image (vLLM) or about 10 minutes for video; confirm GPU is free and disk has the model weights
Training OOM / memory pressure during train or infer	All hardware platforms	Other GPU jobs, residual cache, or workload exceeds available memory	Stop other GPU processes; bring down Streamlit or Jupyter before the next heavy step; flush buffer cache (see note); lower batch size, frames, or LoRA scope
GRPO fine-tuning fails immediately with `ImportError: Found an incompatible version of torchao`	All hardware platforms	The container was built with a newer `peft` that needs `torchao` 0.16 or later, but the base image ships `torchao` 0.13	Pull the latest playbook assets and rebuild the image as in Step 3 (the Dockerfile pins `peft==0.17.1`), then relaunch the container
Memory pressure within capacity	DGX Spark	UMA buffer cache not released	See UMA note below

NOTE

Unified memory (UMA). On hardware platforms with unified memory, GPU and CPU share memory dynamically. Some applications have not yet been updated for UMA, so you may hit memory issues even within capacity. If that happens, manually flush the buffer cache:

Bash

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

For latest known issues, see the documentation linked under Resources for your hardware platform.