---
title: "Nanochat on Dual-Spark"
publisher: "nvidia"
type: "playbook"
updated: "2026-01-29T15:14:00.902Z"
description: "Setup Nanochat on Dual-Spark"
canonical: "https://build.nvidia.com/spark/nanochat-dual-spark.md"
---

# Basic idea

This playbook shows you how to run Andrej Karpathy’s Nanochat on Spark. Nanochat is popularized as being the best ChatGPT that $100 can buy. This playbook makes it possible to train and run Nanochat locally on your dual-Spark setup. 

# What you'll accomplish

You’ll set up a local, end-to-end ChatGPT-like training pipeline, including pre-training, mid-training, post-training, and optional reinforcement learning. You will also be able to chat with your model through a simple web UI. 

# What to know before starting

- Working with Docker containers and GPU passthrough
- Command-line tools for GPU workloads
- Basic understanding of training foundation LLM models

# Prerequisites

- Dual-Spark [setup](../connect-two-sparks/overview.md) with QSFP cable
- Docker installed and accessible to current user
- NVIDIA Container Runtime configured
- Hugging Face [token](https://huggingface.co/settings/tokens) and WandB [API key](https://docs.wandb.ai/models/quickstart#sign-up-and-create-an-api-key)
- Verify GPU access: `nvidia-smi`
- Check Docker GPU integration: `docker run --rm --gpus all nvcr.io/nvidia/pytorch:25.11-py3 nvidia-smi`

# Ancillary files

The reference training scripts can be found in the Nanochat repository [here on GitHub](https://github.com/karpathy/nanochat/)
- [**Dockerfile**](assets/Dockerfile) - Build custom docker to serup the environment
- [**setup.sh**](assets/setup.sh) - Setup the docker image on both Spark machines
- [**speedrun_spark.sh**](assets/speedrun_spark.sh) - Modified version of [speedrun.sh](https://github.com/karpathy/nanochat/blob/master/speedrun.sh) to support distributed training on dual-spark
- [**launch.sh**](assets/launch.sh) - Launch the nanochat training on both Spark machines

# Time & risk

- **Duration**: Upto 5 days depending on model size and number of training stages. 

- **Risks**: 
- Model instantiation and training are memory-intensive
- Modifying hyperparameters such as batch size, model dimensions, or precision settings can increase memory usage and may result in OOM
- Downloading large datasets and storing the trained checkpoints can take up storage space

- **Rollback**: 
- Delete the downloaded dataset and checkpoints from `$HOME/.cache/nanochat`
- Then exit the container environment

## More

- [Instructions](/spark/nanochat-dual-spark/instructions.md)
- [Troubleshooting](/spark/nanochat-dual-spark/troubleshooting.md)