A vision-language-action policy adapted on LIBERO Spatial
| Symptom | Cause | Fix |
|---|---|---|
git clone fails or demo videos are tiny / missing | Git LFS not installed or not initialized | sudo apt-get install -y git-lfs && git lfs install; remove any partial Isaac-GR00T directory; clone again with --recurse-submodules |
GR1, demo_data/gr1.PickNPlace, or scripts do not match the playbook | Wrong branch (for example main tracking a newer GR00T line) | cd Isaac-GR00T && git fetch origin && git checkout n1.6-release && git submodule update --init --recursive |
install_deps.sh not allowed by policy | Script runs sudo apt-get (ffmpeg, libaio-dev, optional CUDA toolkit) and installs uv / syncs .venv | Pre-install the same system packages and CUDA via your IT process; then from the repo root: export PATH="$HOME/.local/bin:$PATH" && uv sync && uv pip install -e .. On aarch64 still need torchcodec or the PyAV patch (Instructions Step 2) |
uv sync appears stuck for hours building flash-attn on aarch64 | Upstream lists prebuilt flash-attn wheels only for linux_x86_64 | Pin flash-attn==2.8.1 plus matching torch 2.10 / torchvision / triton, and add the aarch64 wheel URL under [tool.uv.sources] (see Instructions Step 1c Warning) |
install_deps.sh fails building torchcodec | Missing license confirmation or FFmpeg development libraries | Set I_CONFIRM_THIS_IS_NOT_A_LICENSE_VIOLATION=1 and re-run; if needed install FFmpeg dev packages (libav*-dev, pkg-config, cmake, build-essential, pybind11-dev) then apply Instructions Step 2 |
huggingface-cli download fails with 401 Unauthorized | Missing or invalid HF_TOKEN, or gated-model agreement not accepted | echo $HF_TOKEN && huggingface-cli whoami; export a valid token; accept required agreements on the model page |
Permission denied under ~/.cache/huggingface/hub/... | Cache directory owned by root (common after Docker runs) | export HF_HOME=$HOME/hf_cache_gr00t && mkdir -p "$HF_HOME"; re-export for later steps |
500 Internal Server Error from xet-read-token | Transient Hugging Face xet backend failure | export HF_HUB_DISABLE_XET=1 and retry the download |
externally-managed-environment or packages missing from .venv | PEP 668 / installs not targeting the project venv | source .venv/bin/activate; use uv pip install ... (never sudo pip); recreate .venv with uv sync if needed |
| CUDA out of memory during fine-tuning | Batch size or --tune-llm / --tune-visual exceeds free GPU memory | Lower --global-batch-size (for example 64); check nvidia-smi for other GPU processes |
ptxas-blackwell fatal: Value 'sm_103a' is not defined for option 'gpu-name' | Triton / torch.compile path lacks SM103 support | Prefix inference with TORCHDYNAMO_DISABLE=1 (see Instructions Steps 5 and 8); use the same prefix for eval only if you see the same crash |
ModuleNotFoundError: No module named 'gr00t' | Virtualenv not activated or wrong working directory | source .venv/bin/activate from the Isaac-GR00T repo root |
NotImplementedError in get_frames_by_indices when backend is pyav | Stock n1.6-release lacks the pyav indexed-frame branch | Apply the playbook PyAV patch and uv pip install av (Instructions Step 2; assets/patches/README.md) |
| Training “hangs” — low GPU utilization, no traceback, very slow steps | Fallback to per-frame ffmpeg subprocess decoding for AV1 LIBERO clips | Apply the PyAV patch + uv pip install av; optionally raise --dataloader-num-workers (for example 8) |
Video decoding errors / torchcodec not found | Preferred video backend missing or build failed | Prefer PyAV patch + av; to build torchcodec manually into .venv see detailed notes below |
| Training loss is not decreasing | Early run, wrong modality, or LR too low for a short smoke test | Verify meta/modality.json, confirm --embodiment-tag LIBERO_PANDA, try --learning-rate 5e-4 on short runs |
nvidia-smi shows the wrong GPU | Multi-GPU host; CUDA_VISIBLE_DEVICES points at a secondary card | nvidia-smi --query-gpu=index,name --format=csv,noheader then set CUDA_VISIBLE_DEVICES=<index> |
| OpenCV or decord cannot decode LIBERO AV1 | OpenCV often fails on AV1; decord may lack a compatible wheel | Use the PyAV patch path documented in this playbook |
Prefer the PyAV patch + av path for LIBERO. If you must build torchcodec into .venv manually (aarch64), with FFmpeg development packages installed:
# Run this from inside the Isaac-GR00T repo root (the directory that
# contains .venv). Capture its absolute path BEFORE changing directories
# so we can still reach the virtualenv after cd'ing into /tmp/torchcodec.
GR00T_ROOT="$(pwd)"
# Sanity check — the virtualenv interpreter must already exist.
test -x "$GR00T_ROOT/.venv/bin/python" || { echo "Not in Isaac-GR00T root (missing .venv/bin/python)"; }
# Clone the torchcodec source into /tmp/torchcodec (skip if already cloned).
git clone https://github.com/pytorch/torchcodec.git /tmp/torchcodec
cd /tmp/torchcodec
# Build torchcodec into the Isaac-GR00T virtualenv using the absolute
# path captured above (do NOT use the relative ".venv/bin/python" here —
# the current directory is /tmp/torchcodec, which has no .venv).
I_CONFIRM_THIS_IS_NOT_A_LICENSE_VIOLATION=1 ENABLE_CUDA=1 \
uv pip install --python "$GR00T_ROOT/.venv/bin/python" . --no-build-isolation
CUDA-enabled builds can fail when system FFmpeg or CUDA does not match torchcodec expectations — in that case use the PyAV patch instead.
For latest known issues, see the documentation linked under Resources for your hardware platform.