---
title: "Serve Models with llama.cpp — Overview"
canonical: "https://build.nvidia.com/playbooks/llama-cpp/overview.md"
---

# Basic idea

[llama.cpp](https://github.com/ggml-org/llama.cpp) is a lightweight C/C++ inference stack for large language models. You build it with CUDA so it fully utilizes the GPU on your hardware platform, then load GGUF weights and expose chat through `llama-server`’s OpenAI-compatible HTTP API.

This playbook walks through that stack end to end using MTP-enabled **Qwen3.6-35B-A3B** as the hands-on example. Any GGUF checkpoint that fits in available memory works with the same workflow; commands are in the **Instructions** tab.

# What you'll accomplish

You'll build llama.cpp with CUDA, download a **Qwen3.6-35B-A3B** GGUF checkpoint, and run **`llama-server`** with GPU offload on your **hardware platform**. You get:

- Local inference through llama.cpp (no separate Python inference framework required)
- An OpenAI-compatible `/v1/chat/completions` endpoint for tools and apps
- A concrete validation that the **Qwen3.6-35B-A3B** example runs with MTP support

# What to know before starting

**Required:**

- Basic familiarity with the Linux command line
- Understanding of git and building from source with CMake
- Basic knowledge of REST APIs and cURL for testing

**Optional:**

- Familiarity with Hugging Face model downloads and GGUF quantization tags

# Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

| Hardware platform | OS | Memory | Recommended default local settings | Multi-node capable hardware |
| :---- | :---- | :---- | :---- | :---- |
| **DGX Spark** | DGX OS | 128 GB Unified Memory | Build from source (`CMAKE_CUDA_ARCHITECTURES=121a-real`); example model `unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL` | — |

# Prerequisites

**Hardware requirements**

- Supported hardware platform — see Supported hardware platforms matrix above
- Sufficient memory for the model and KV cache (about 30 GB free for the example model)
- At least ~40 GB free disk for the example download plus build artifacts (more if you keep multiple GGUFs)

**Software requirements**

- Git: `git --version`
- CMake (3.18+): `cmake --version`
- CUDA Toolkit: `nvcc --version`
- Network access to GitHub and Hugging Face

# Find model recipes

llama.cpp loads models in **GGUF** format. This playbook’s walkthrough uses the MTP-enabled Qwen3.6-35B-A3B quant below. Browse other GGUF checkpoints on Hugging Face and confirm they fit in available memory on your hardware platform.

| Hardware platform | Example / more models |
| ----------------- | --------------------- |
| **DGX Spark** | Walkthrough: [`unsloth/Qwen3.6-35B-A3B-MTP-GGUF`](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) · Explore GGUF models on [Hugging Face](https://huggingface.co/models?library=gguf) |

Use the **Instructions** tab for the base build-and-serve workflow. For agentic workloads with the recommended Qwen3.6 checkpoint, see the **Agent-ready Models** tab.

> [!NOTE]
> **Memory determines what you can run.** Any GGUF that fits in available memory can be served with the same `llama-server` workflow. If a model is not listed for your hardware platform, check memory headroom and try the base configuration in **Instructions**.

# Time & risk

- **Estimated time:** 30 MIN (longer on first run due to the example GGUF download — about 21 GB for the default quant)
- **Risk level:** Low
- The build is local to your clone; Step 1 installs a few system packages with `apt`
- Large GGUF downloads may fail or stall due to network issues
- **Rollback:** Remove the `llama.cpp` clone and the model directory under `~/.cache/huggingface/hub/` to reclaim disk space
- **Last Updated:** 10/09/2026
- Fixed the `llama-server` launch path, raised the CMake minimum to 3.18, corrected the model download size, and updated the sample API response for the Qwen3.6 thinking model