---
title: "Run models with llama.cpp on DGX Spark"
publisher: "nvidia"
type: "playbook"
updated: "2026-04-02T18:27:46.743Z"
description: "Build llama.cpp with CUDA and serve models via an OpenAI-compatible API"
canonical: "https://build.nvidia.com/spark/llama-cpp.md"
---

# Basic idea

[llama.cpp](https://github.com/ggml-org/llama.cpp) is a lightweight C/C++ inference stack for large language models. You build it with CUDA so it fully utilizes the DGX Spark GB10 GPU, then load GGUF weights and expose chat through `llama-server`’s OpenAI-compatible HTTP API.

This playbook walks through that stack end to end using MTP-enabled **Qwen3.6-35B-A3B** as the hands-on example. Checkpoint choices and paths for all supported models are summarized in the matrix below; commands are in the instructions.

# What you'll accomplish

You will build llama.cpp with CUDA for GB10, download a **Qwen3.6-35B-A3B** checkpoint, and run **`llama-server`** with GPU offload. You get:

- Local inference through llama.cpp (no separate Python inference framework required)  
- An OpenAI-compatible `/v1/chat/completions` endpoint for tools and apps  
- A concrete validation that the **Qwen3.6-35B-A3B** example runs on this stack on DGX Spark with MTP support.

# What to know before starting

- Basic familiarity with Linux command line and terminal commands  
- Understanding of git and building from source with CMake  
- Basic knowledge of REST APIs and cURL for testing

# Prerequisites

**Hardware requirements**

- NVIDIA DGX Spark with GB10 GPU  
- Sufficient unified memory for the model and the KV-Cache being utilized (about 30GB free RAM for the model in the example)  
- At least **\~40GB** free disk for the example download plus build artifacts (more if you keep multiple GGUFs)

**Software requirements**

- NVIDIA DGX OS  
- Git: `git --version`  
- CMake (3.14+): `cmake --version`  
- CUDA Toolkit: `nvcc --version`  
- Network access to GitHub and Hugging Face

# Model support matrix

DGX Spark supports any GGUF  format model checkpoint with llama.cpp, as long as the system has memory available to host and run the checkpoint.

# Time & risk

* **Estimated time:** About 30 minutes, plus downloading the example GGUF (\~35GB order of magnitude for the default quant)  
* **Risk level:** Low — build is local to your clone; no system-wide installs required for the steps below  
* **Rollback:** Remove the `llama.cpp` clone and the model directory under `~/.cache/huggingface/hub/` to reclaim disk space  
* **Last updated:** 06/03/2026  
* Walkthrough now uses Qwen3.6-35B-A3B as an example

## More

- [Instructions](/spark/llama-cpp/instructions.md)
- [Troubleshooting](/spark/llama-cpp/troubleshooting.md)