Train a Chat Model with NanoChat
12 HOURS
Build a ChatGPT-style LLM end-to-end — tokenizer, pretraining, SFT — then chat via web UI or CLI
Common issues
The Hardware platform column shows where an issue is most relevant. "All hardware platforms" applies to every supported platform.
| Symptom | Hardware platform | Cause | Fix |
|---|---|---|---|
| "permission denied" when running docker | All hardware platforms | User not in docker group | Run sudo usermod -aG docker $USER && newgrp docker |
WANDB_API_KEY is not set or HF_TOKEN is not set | All hardware platforms | Required env vars not exported before launch | export WANDB_API_KEY=<key> and export HF_TOKEN=<token> in the same shell, then re-run launch |
RuntimeError: CUDA out of memory | All hardware platforms | Batch size or model depth too large | Reduce --device-batch-size / --device_batch_size (try 64 → 32 → 16 → 8) or lower --depth in the speedrun script; re-run setup then launch |
| Container fails to start / GPU not visible | All hardware platforms | NVIDIA Container Toolkit not configured, or GPU in use | Test docker run --rm --gpus all nvcr.io/nvidia/pytorch:<tag> nvidia-smi; ensure no other GPU containers are running; check docker logs |
nanochat image not found | All hardware platforms | Setup not run or Docker build failed | From assets/, run ./setup.sh and confirm with docker images | grep nanochat |
| Training hangs at dataset download | All hardware platforms | Network issue downloading shards | Check connectivity; downloads can take a long time — restart launch if it stalls indefinitely |
Disk full / No space left on device | All hardware platforms | Dataset + checkpoints exhausted disk | Ensure ~50 GB free before training; docker system prune; remove old caches under ~/.cache/nanochat or ./nanochat_cache |
| Web UI not reachable on port 8000 | All hardware platforms | Server not running, port blocked, or missing SSH tunnel | Confirm chat_web is running; ssh -L 8000:localhost:8000 user@<HOST_IP> if remote; allow port 8000 if firewalled |
| Model runs on wrong GPU | multi-GPU hosts | Default GPU selection | Pin with GPU_DEVICE='"device=N"' ./launch.sh (N from nvidia-smi) |
Getting additional help
- nanochat issues: https://github.com/karpathy/nanochat/issues
- Container logs:
docker logs <container_id> - System resources:
htopandnvidia-smi - NVIDIA Developer Forums: https://forums.developer.nvidia.com/