Hugging Face model training from single-GPU to multi-node jobs
Confirm your hardware platform meets the prerequisites for NeMo AutoModel. Run these checks on the host to confirm CUDA, Python, GPU access, memory, and Docker.
# Verify CUDA installation
nvcc --version
# Check Python version (3.10+ required)
python3 --version
# Verify GPU accessibility
nvidia-smi
# Check available system memory
free -h
# Docker access
docker ps
If docker ps returns a permission denied error (for example, permission denied while trying to connect to the Docker daemon socket), complete Step 2. Otherwise continue to Step 3.
To manage containers without sudo, add your user to the docker group. If you skip this step, run Docker commands with sudo.
sudo usermod -aG docker $USER
newgrp docker
Open a new terminal (or continue after newgrp) and confirm access:
docker ps
Pull the 26.08 container for the runtime dependencies. The Spark-specific recipes and memory fixes landed after the image was built, so clone NeMo AutoModel and pin it to the merge commit for PR 3766.
docker pull nvcr.io/nvidia/nemo-automodel:26.08
AUTOMODEL_COMMIT=3b5663a6fdf0fee03abbe82e431ca3a7ba6dd8e2
git clone https://github.com/NVIDIA-NeMo/Automodel.git "$HOME/Automodel"
git -C "$HOME/Automodel" checkout --detach "$AUTOMODEL_COMMIT"
git -C "$HOME/Automodel" rev-parse HEAD
The last command should print 3b5663a6fdf0fee03abbe82e431ca3a7ba6dd8e2.
Launch an interactive container with GPU access and the 64 GB limit used for the lower-memory Spark validation. Mount the pinned checkout over the source included in the image; this also keeps checkpoints on the host after the container exits. The --rm flag removes the container when you exit.
docker run \
--gpus all \
--ipc=host \
--memory=64g \
--memory-swap=64g \
--ulimit memlock=-1 \
-it --ulimit stack=67108864 \
-e PYTHONDONTWRITEBYTECODE=1 \
-v "$HOME/Automodel:/opt/Automodel" \
-w /opt/Automodel \
--entrypoint /usr/bin/bash \
--rm nvcr.io/nvidia/nemo-automodel:26.08
Review the pre-configured training recipes for different model types and training scenarios.
# Navigate to /opt/Automodel
cd /opt/Automodel
# List RTX Spark fine-tuning recipes
find examples/llm_finetune -name '*_rtx_spark.yaml' -print
# View example recipe configuration
sed -n '1,40p' examples/llm_finetune/llama3_1/llama3_1_8b_squad_peft_rtx_spark.yaml
The following commands show parameter-efficient fine-tuning (PEFT) with the LoRA and QLoRA recipes validated for Spark unified memory.
First, export your Hugging Face token so gated models can be downloaded.
export HF_TOKEN=<your_huggingface_token>
NOTE
Replace <your_huggingface_token> with your personal Hugging Face access token. A valid token is required to download any gated model. License acceptance and access approval must belong to the same Hugging Face account that issued the token; exporting the token does not request or grant gated-model access.
The same steps apply for any other gated model you use: visit its model card on Hugging Face, request access, accept the license, and wait for approval.
IMPORTANT
Keep the batch size, packed sequence size, attention, activation checkpointing, and checkpoint-loading settings from these _rtx_spark recipes unless you have validated different settings on your hardware.
Llama 3.1 8B LoRA fine-tuning example:
Run a basic fine-tuning example to validate the setup. This demonstrates parameter-efficient fine-tuning with a model suitable for testing. The examples below use YAML for configuration; parameter overrides are passed as command-line arguments.
cd /opt/Automodel
automodel examples/llm_finetune/llama3_1/llama3_1_8b_squad_peft_rtx_spark.yaml \
--nproc-per-node 1 \
--step_scheduler.max_steps 20
The recipe selects meta-llama/Llama-3.1-8B and the validated Spark memory settings. The command-line arguments run it on one GPU for 20 training steps.
Llama 3.3 70B QLoRA fine-tuning example:
Use QLoRA to fine-tune large models in a memory-efficient manner.
cd /opt/Automodel
automodel examples/llm_finetune/llama3_3/llama_3_3_70b_instruct_squad_peft_qlora_rtx_spark.yaml \
--nproc-per-node 1 \
--step_scheduler.max_steps 20
The recipe selects meta-llama/Llama-3.3-70B-Instruct, loads it in 4-bit mode, and uses the validated Spark memory settings. The command-line arguments run it on one GPU for 20 training steps.
Qwen3 8B LoRA fine-tuning example:
Run the following command to fine-tune Qwen3-8B with LoRA:
cd /opt/Automodel
automodel examples/llm_finetune/qwen/qwen3_8b_squad_rtx_spark.yaml \
--nproc-per-node 1 \
--step_scheduler.max_steps 20
The recipe selects Qwen/Qwen3-8B and the validated Spark memory settings. The command-line arguments run it on one GPU for 20 training steps.
Validate the fine-tuned model by inspecting artifacts in the checkpoint directory.
# Inspect logs and checkpoint output.
# LATEST is a symlink pointing to the latest checkpoint.
# Below is an example of expected output (username and domain-users are placeholders).
ls -lah checkpoints/LATEST/
# $ ls -lah checkpoints/LATEST/
# total 32K
# drwxr-xr-x 6 username domain-users 4.0K Oct 16 22:33 .
# drwxr-xr-x 4 username domain-users 4.0K Oct 16 22:33 ..
# -rw-r--r-- 1 username domain-users 1.6K Oct 16 22:33 config.yaml
# drwxr-xr-x 2 username domain-users 4.0K Oct 16 22:33 dataloader
# drwxr-xr-x 2 username domain-users 4.0K Oct 16 22:33 model
# drwxr-xr-x 2 username domain-users 4.0K Oct 16 22:33 optim
# drwxr-xr-x 2 username domain-users 4.0K Oct 16 22:33 rng
# -rw-r--r-- 1 username domain-users 1.3K Oct 16 22:33 step_scheduler.pt
The container was launched with the --rm flag, so it is automatically removed when you exit. The mounted source and checkpoints remain in $HOME/Automodel. To reclaim disk space used by the Docker image, run:
WARNING
This will remove the NeMo AutoModel image. You will need to pull it again if you want to use it later.
docker rmi nvcr.io/nvidia/nemo-automodel:26.08
Remove $HOME/Automodel only after copying any checkpoints you want to keep.
Publish your fine-tuned model checkpoint on Hugging Face Hub.
NOTE
This is an optional step and is not required for using the fine-tuned model.
It is useful if you want to share your fine-tuned model with others or use it in other projects.
To use the Hugging Face CLI, install it with pip install huggingface_hub.
For more information, see the Hugging Face CLI documentation.
TIP
You can use the hf command to upload the fine-tuned model checkpoint to Hugging Face Hub.
For more information, see the Hugging Face CLI documentation.
# Publish under the namespace <your_huggingface_username>/my-cool-model; adjust the name as needed.
hf upload my-cool-model checkpoints/LATEST/model
TIP
The above command can fail if you don't have write permissions to the Hugging Face Hub with the HF_TOKEN you used.
Sample error message:
user@host:/opt/Automodel$ hf upload my-cool-model checkpoints/LATEST/model
Traceback (most recent call last):
File "/home/user/.local/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 409, in hf_raise_for_status
response.raise_for_status()
File "/home/user/.local/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/repos/create
To fix this, create an access token with write permissions. See the Hugging Face guide here.
Begin using NeMo AutoModel for your specific fine-tuning tasks. Start with the provided recipes and customize them for your model and dataset.
# Copy a Spark recipe for customization
cp examples/llm_finetune/llama3_1/llama3_1_8b_squad_peft_rtx_spark.yaml my_custom_recipe.yaml
# Edit configuration for your specific model and data, then run:
automodel my_custom_recipe.yaml --nproc-per-node 1
Explore the NeMo AutoModel GitHub repository for more recipes, documentation, and community examples. Consider setting up custom datasets and experimenting with different model architectures.