---
title: "Fine-Tune Isaac GR00T for Robot Skills — Overview"
canonical: "https://build.nvidia.com/playbooks/gr00t/overview.md"
---

# Basic idea

NVIDIA Isaac GR00T N1.6 is a 3-billion-parameter open vision-language-action (VLA) foundation model for generalist humanoid robot skills. It combines a Cosmos-family vision-language backbone with a 32-layer Diffusion Transformer (DiT) action head that denoises continuous robot actions from multimodal input — language instructions and camera images. The model is pre-trained on a large mixture of robot demonstration data, then adapted to specific embodiments and tasks through fine-tuning.

High-level architecture (VLM + DiT action head), as in the upstream Isaac GR00T repo:

![GR00T N1.6 reference architecture](./assets/GR00T-reference-arch-diagram.png)

*Source: [NVIDIA Isaac GR00T — `media/GR00T-reference-arch-diagram.png`](https://github.com/NVIDIA/Isaac-GR00T/blob/n1.6-release/media/GR00T-reference-arch-diagram.png). If the local image above is missing, the upstream copy is at `https://raw.githubusercontent.com/NVIDIA/Isaac-GR00T/n1.6-release/media/GR00T-reference-arch-diagram.png`.*

In this playbook you fine-tune GR00T N1.6 on the **LIBERO Spatial** benchmark on your **hardware platform**. Large unified GPU memory supports a high **global batch size (128)** on a single GPU, which improves training throughput compared with typical 24–80 GB cards.

# LIBERO Spatial (what you are fine-tuning on)

**LIBERO Spatial** is part of the [LIBERO](https://libero-project.github.io/main.html) suite of simulated tabletop manipulation benchmarks. The **spatial** split emphasizes **where** objects need to be placed: tasks such as putting a bowl on a **stove burner** vs a **plate**, placing utensils in a **mug** vs next to it, or moving objects to **left/right/front** targets on the table. Episodes include third-person RGB video, proprioceptive state, language instructions, and continuous end-effector actions in a consistent LeRobot v2 layout. Understanding these constraints helps when you read training logs or open-loop evaluation plots.

# What kind of fine-tuning this playbook uses

This playbook runs the **default Isaac GR00T fine-tuning recipe** from `launch_finetune.py`: **not** full-model weight updates of the entire 3B VLM. In the stock configuration, training focuses on the **action head (DiT)** and **projector / adapter paths** that map observations into the action model, with strong **state dropout** and **color jitter** so the policy leans on vision. Optional flags such as `--tune-llm` or `--tune-visual` (mentioned under Next steps) trade compute and memory for updating more of the backbone. **LoRA** is not the default here; if your team uses LoRA or other PEFT variants, treat that as a separate configuration branch from this playbook.

# What you'll accomplish

You'll fine-tune and evaluate Isaac GR00T N1.6 on your **hardware platform**:

- Check out the **`n1.6-release`** branch of Isaac GR00T so commands, embodiment tags, and `demo_data/` match GR00T **N1.6**
- Set up the environment with `uv` (project-local `.venv`) and understand what the optional `install_deps.sh` script changes on the system
- Apply the recommended **PyAV `get_frames_by_indices` patch** when `torchcodec` is unavailable so LIBERO **AV1** video decoding does not stall on an **ffmpeg** subprocess fallback
- Verify the base model, fine-tune on LIBERO Spatial at batch size **128**, run open-loop evaluation, and measure inference latency

# What to know before starting

**Required:**

- Familiarity with Python virtual environments (`source .venv/bin/activate`)
- Familiarity with PyTorch training concepts (batch size, loss, checkpoints)
- Comfort running commands that may use **`sudo`** for system packages (or use the documented user-space alternative)

**Optional:**

- Basic robot manipulation vocabulary (trajectories, observations, actions)

# Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

| Hardware platform | OS | Memory | Recommended default local settings | Multi-node capable hardware |
| :---- | :---- | :---- | :---- | :---- |
| **DGX Station** | DGX OS (Linux) | ~284 GB HBM3e (GB300) | Isaac GR00T `n1.6-release`; `uv` + PyAV patch; global batch size **128** | — |

# Prerequisites

**Hardware requirements**

- Supported hardware platform — see Supported hardware platforms matrix above
- At least **~30 GB** free disk for `.venv`, checkpoints, and the LIBERO download

**Software requirements**

- CUDA toolkit usable by PyTorch: `nvcc --version` should show **CUDA 12.8+** (often already under `/usr/local/cuda`)
- **Git** and **Git LFS** (`git lfs version`) — LFS is required for some demo assets and submodules; install with `sudo apt-get install -y git-lfs` then `git lfs install` if missing
- Hugging Face account and **HF_TOKEN** for model and dataset downloads
- Network access to Hugging Face, GitHub, and PyPI

# Ancillary files

All required assets can be found [in this playbook repository](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/playbook-gr00t/).

- `assets/GR00T-reference-arch-diagram.png` — Architecture diagram shown above
- `assets/patches/001-pyav-get-frames-by-indices.patch` — PyAV indexed-frame reader for LIBERO video
- `assets/patches/README.md` — When and how to apply the patch
- `assets/patches/video_utils_*.py` — Reference snippets for the patched `video_utils` path

# Time & risk

- **Estimated time:** 45 MIN end-to-end when the video backend is healthy (setup, downloads, ~20–25 min training at 2000 steps, eval and inference). With the PyAV fallback and no `torchcodec`, full 2000-step training can take several hours — see Instructions.
- **Risk level:** Medium
- `scripts/deployment/dgpu/install_deps.sh` performs **system-level** `apt` operations and may install the **CUDA 12.8 toolkit** if `/usr/local/cuda` is absent
- Model and dataset downloads require Hugging Face authentication
- Video decoding fallbacks can stall training if the PyAV patch is skipped
- **Rollback:** Remove the cloned `Isaac-GR00T` directory and optionally `rm -rf ~/.local/share/uv` if you want to reclaim `uv` caches. Reverting `apt`-installed packages is a separate admin task; the playbook does not uninstall them automatically.
- **Last Updated:** 08/03/2026
- Fine-tune Isaac GR00T N1.6 on LIBERO Spatial with setup, PyAV video patch, evaluation, and inference steps on supported hardware platforms