Serve Models with llama.cpp

Basic idea

llama.cpp is a lightweight C/C++ inference stack for large language models. You build it with CUDA so it fully utilizes the GPU on your hardware platform, then load GGUF weights and expose chat through llama-server’s OpenAI-compatible HTTP API.

This playbook walks through that stack end to end using MTP-enabled Qwen3.6-35B-A3B as the hands-on example. Any GGUF checkpoint that fits in available memory works with the same workflow; commands are in the Instructions tab.

What you'll accomplish

You'll build llama.cpp with CUDA, download a Qwen3.6-35B-A3B GGUF checkpoint, and run llama-server with GPU offload on your hardware platform. You get:

Local inference through llama.cpp (no separate Python inference framework required)
An OpenAI-compatible /v1/chat/completions endpoint for tools and apps
A concrete validation that the Qwen3.6-35B-A3B example runs with MTP support

What to know before starting

Required:

Basic familiarity with the Linux command line
Understanding of git and building from source with CMake
Basic knowledge of REST APIs and cURL for testing

Optional:

Familiarity with Hugging Face model downloads and GGUF quantization tags

Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

Hardware platform	OS	Memory	Recommended default local settings	Multi-node capable hardware
DGX Spark	DGX OS	128 GB Unified Memory	Build from source (`CMAKE_CUDA_ARCHITECTURES=121a-real`); example model `unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL`	—

Prerequisites

Hardware requirements

Supported hardware platform — see Supported hardware platforms matrix above
Sufficient memory for the model and KV cache (about 30 GB free for the example model)
At least ~40 GB free disk for the example download plus build artifacts (more if you keep multiple GGUFs)

Software requirements

Git: git --version
CMake (3.18+): cmake --version
CUDA Toolkit: nvcc --version
Network access to GitHub and Hugging Face

Find model recipes

llama.cpp loads models in GGUF format. This playbook’s walkthrough uses the MTP-enabled Qwen3.6-35B-A3B quant below. Browse other GGUF checkpoints on Hugging Face and confirm they fit in available memory on your hardware platform.

Hardware platform	Example / more models
DGX Spark	Walkthrough: `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` · Explore GGUF models on Hugging Face

Use the Instructions tab for the base build-and-serve workflow. For agentic workloads with the recommended Qwen3.6 checkpoint, see the Agent-ready Models tab.

NOTE

Memory determines what you can run. Any GGUF that fits in available memory can be served with the same llama-server workflow. If a model is not listed for your hardware platform, check memory headroom and try the base configuration in Instructions.

Time & risk

Estimated time: 30 MIN (longer on first run due to the example GGUF download — about 21 GB for the default quant)
Risk level: Low
- The build is local to your clone; Step 1 installs a few system packages with apt
- Large GGUF downloads may fail or stall due to network issues
Rollback: Remove the llama.cpp clone and the model directory under ~/.cache/huggingface/hub/ to reclaim disk space
Last Updated: 10/09/2026
- Fixed the llama-server launch path, raised the CMake minimum to 3.18, corrected the model download size, and updated the sample API response for the Qwen3.6 thinking model