---
title: "Fine-Tune Isaac GR00T for Robot Skills — Troubleshooting"
canonical: "https://build.nvidia.com/playbooks/gr00t/troubleshooting.md"
---

| Symptom | Cause | Fix |
|---------|-------|-----|
| `git clone` fails or demo videos are tiny / missing | Git LFS not installed or not initialized | `sudo apt-get install -y git-lfs && git lfs install`; remove any partial `Isaac-GR00T` directory; clone again with `--recurse-submodules` |
| `GR1`, `demo_data/gr1.PickNPlace`, or scripts do not match the playbook | Wrong branch (for example `main` tracking a newer GR00T line) | `cd Isaac-GR00T && git fetch origin && git checkout n1.6-release && git submodule update --init --recursive` |
| `install_deps.sh` not allowed by policy | Script runs `sudo apt-get` (ffmpeg, libaio-dev, optional CUDA toolkit) and installs `uv` / syncs `.venv` | Pre-install the same system packages and CUDA via your IT process; then from the repo root: `export PATH="$HOME/.local/bin:$PATH" && uv sync && uv pip install -e .`. On aarch64 still need `torchcodec` or the PyAV patch (Instructions Step 2) |
| `uv sync` appears stuck for hours building `flash-attn` on aarch64 | Upstream lists prebuilt `flash-attn` wheels only for `linux_x86_64` | Pin `flash-attn==2.8.1` plus matching torch 2.10 / torchvision / triton, and add the aarch64 wheel URL under `[tool.uv.sources]` (see Instructions Step 1c Warning) |
| `install_deps.sh` fails building torchcodec | Missing license confirmation or FFmpeg development libraries | Set `I_CONFIRM_THIS_IS_NOT_A_LICENSE_VIOLATION=1` and re-run; if needed install FFmpeg dev packages (`libav*-dev`, `pkg-config`, `cmake`, `build-essential`, `pybind11-dev`) then apply Instructions Step 2 |
| `huggingface-cli download` fails with 401 Unauthorized | Missing or invalid `HF_TOKEN`, or gated-model agreement not accepted | `echo $HF_TOKEN && huggingface-cli whoami`; export a valid token; accept required agreements on the model page |
| `Permission denied` under `~/.cache/huggingface/hub/...` | Cache directory owned by root (common after Docker runs) | `export HF_HOME=$HOME/hf_cache_gr00t && mkdir -p "$HF_HOME"`; re-export for later steps |
| `500 Internal Server Error` from `xet-read-token` | Transient Hugging Face xet backend failure | `export HF_HUB_DISABLE_XET=1` and retry the download |
| `externally-managed-environment` or packages missing from `.venv` | PEP 668 / installs not targeting the project venv | `source .venv/bin/activate`; use `uv pip install ...` (never `sudo pip`); recreate `.venv` with `uv sync` if needed |
| CUDA out of memory during fine-tuning | Batch size or `--tune-llm` / `--tune-visual` exceeds free GPU memory | Lower `--global-batch-size` (for example `64`); check `nvidia-smi` for other GPU processes |
| `ptxas-blackwell fatal: Value 'sm_103a' is not defined for option 'gpu-name'` | Triton / `torch.compile` path lacks SM103 support | Prefix inference with `TORCHDYNAMO_DISABLE=1` (see Instructions Steps 5 and 8); use the same prefix for eval only if you see the same crash |
| `ModuleNotFoundError: No module named 'gr00t'` | Virtualenv not activated or wrong working directory | `source .venv/bin/activate` from the Isaac-GR00T repo root |
| `NotImplementedError` in `get_frames_by_indices` when backend is `pyav` | Stock `n1.6-release` lacks the `pyav` indexed-frame branch | Apply the playbook PyAV patch and `uv pip install av` (Instructions Step 2; `assets/patches/README.md`) |
| Training “hangs” — low GPU utilization, no traceback, very slow steps | Fallback to per-frame `ffmpeg` subprocess decoding for AV1 LIBERO clips | Apply the PyAV patch + `uv pip install av`; optionally raise `--dataloader-num-workers` (for example `8`) |
| Video decoding errors / `torchcodec` not found | Preferred video backend missing or build failed | Prefer PyAV patch + `av`; to build torchcodec manually into `.venv` see detailed notes below |
| Training loss is not decreasing | Early run, wrong modality, or LR too low for a short smoke test | Verify `meta/modality.json`, confirm `--embodiment-tag LIBERO_PANDA`, try `--learning-rate 5e-4` on short runs |
| `nvidia-smi` shows the wrong GPU | Multi-GPU host; `CUDA_VISIBLE_DEVICES` points at a secondary card | `nvidia-smi --query-gpu=index,name --format=csv,noheader` then set `CUDA_VISIBLE_DEVICES=<index>` |
| OpenCV or decord cannot decode LIBERO AV1 | OpenCV often fails on AV1; decord may lack a compatible wheel | Use the PyAV patch path documented in this playbook |

## Manual torchcodec build (optional)

Prefer the **PyAV patch + `av`** path for LIBERO. If you must build **torchcodec** into `.venv` manually (aarch64), with FFmpeg development packages installed:

```bash
# Run this from inside the Isaac-GR00T repo root (the directory that
# contains .venv). Capture its absolute path BEFORE changing directories
# so we can still reach the virtualenv after cd'ing into /tmp/torchcodec.
GR00T_ROOT="$(pwd)"

# Sanity check — the virtualenv interpreter must already exist.
test -x "$GR00T_ROOT/.venv/bin/python" || { echo "Not in Isaac-GR00T root (missing .venv/bin/python)"; }

# Clone the torchcodec source into /tmp/torchcodec (skip if already cloned).
git clone https://github.com/pytorch/torchcodec.git /tmp/torchcodec
cd /tmp/torchcodec

# Build torchcodec into the Isaac-GR00T virtualenv using the absolute
# path captured above (do NOT use the relative ".venv/bin/python" here —
# the current directory is /tmp/torchcodec, which has no .venv).
I_CONFIRM_THIS_IS_NOT_A_LICENSE_VIOLATION=1 ENABLE_CUDA=1 \
uv pip install --python "$GR00T_ROOT/.venv/bin/python" . --no-build-isolation
```

CUDA-enabled builds can fail when system FFmpeg or CUDA does not match torchcodec expectations — in that case use the **PyAV patch** instead.

For latest known issues, see the documentation linked under **Resources** for your hardware platform.