Use https://build.nvidia.com/playbooks/llama-cpp.md and complete this playbook on this machine: Serve Models with llama.cpp. Fetch that .md URL directly; the non-.md page is a JavaScript shell. When finished, state clearly whether the playbook succeeded, and give the evidence: what you installed, where, and the output of a command proving it works (a version string, a container listing, or an HTTP status).
NOTE
This has been tested using OpenCode and Qwen 3.8 35B A3B NVFP4. Performance and outcome may vary based on agent and model selection.
Update the package metadata and install the required dependencies:
sudo apt update
sudo apt install -y git clang cmake libcurl4-openssl-dev libssl-dev
Clone upstream llama.cpp — the framework you are building:
git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp
cd ~/llama.cpp
Configure CMake with CUDA so GGML’s CUDA backend matches your hardware platform. For the platforms in this playbook, use CUDA architecture 121a-real:
cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CURL=ON -DGGML_RPC=ON -DCMAKE_CUDA_ARCHITECTURES=121a-real
cmake --build build --config Release --target llama-server -j
The build usually takes on the order of 5–10 minutes. When it finishes, llama-server appears under build/bin/.
llama.cpp loads models in GGUF format. This playbook uses the Q4_K_XL checkpoint from unsloth/Qwen3.6-35B-A3B-MTP-GGUF, which provides a good balance between quality and speed on the supported hardware platforms.
From the ~/llama.cpp directory you cloned in Step 2, launch the OpenAI-compatible server with GPU offload. It loads the model from Hugging Face first if it has not been downloaded before or if there are updates.
All models are saved in the default Hugging Face cache directory under ~/.cache/huggingface/hub. For example, this model is saved into ~/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-MTP-GGUF.
It also automatically loads an mmproj file to enable vision capabilities if supported by the model. By default, llama-server tries to fit the full model context with the ability to serve 4 concurrent requests, and adjusts parameters automatically if needed.
./build/bin/llama-server \
-hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
--host 0.0.0.0 \
--port 30000
To run with MTP speculative decoding, provide additional parameters as shown below. MTP requires a compatible model, like unsloth/Qwen3.6-35B-A3B-MTP-GGUF used in this example. The following example also sets the preserve_thinking flag so Qwen models can use interleaved thinking by preserving prior thinking blocks in the history — useful for agentic workflows. For a dedicated agentic launch path, see the Agent-ready Models tab.
./build/bin/llama-server \
-hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
--host 0.0.0.0 \
--port 30000 \
--chat-template-kwargs '{"preserve_thinking": true}' \
--spec-type draft-mtp \
--spec-draft-n-max 3
Parameters (short):
--host / --port: bind address and port for the HTTP API--n-gpu-layers: number of model layers to offload to the GPU; llama.cpp normally chooses this for you, so set it explicitly (for example --n-gpu-layers 99) only if nvidia-smi shows the model is not running on the GPU--chat-template-kwargs: additional params for the JSON template parser; must be a valid JSON object string--spec-type: comma-separated list of speculative decoding types (default: none; most MTP-compatible models use draft-mtp — check the model card)--spec-draft-n-max: number of tokens to draft for speculative decoding (default: 3)You should see log lines similar to:
0.14.322.968 I srv load_model: speculative decoding context initialized
0.14.322.970 I slot load_model: id 0 | task -1 | new slot, n_ctx = 262144
0.14.322.972 I slot load_model: id 1 | task -1 | new slot, n_ctx = 262144
0.14.322.972 I slot load_model: id 2 | task -1 | new slot, n_ctx = 262144
0.14.322.973 I slot load_model: id 3 | task -1 | new slot, n_ctx = 262144
0.14.323.063 I srv load_model: prompt cache is enabled, size limit: 8192 MiB
...
0.14.342.935 I srv llama_server: model loaded
0.14.342.939 I srv llama_server: server is listening on http://0.0.0.0:30000
0.14.342.944 I srv update_slots: all slots are idle
Keep this terminal open while testing. Large GGUFs can take a minute or more to load, and the initial model download can take a while if the model is not downloaded yet. You will see a progress bar when the model is being downloaded.
The server is ready to accept connections on port 30000 after you see the server is listening message (see Troubleshooting if curl reports connection refused).
Before sending requests, confirm the server is ready (large models can take several minutes to load):
timeout 900 bash -c 'until curl -sf http://127.0.0.1:30000/health > /dev/null 2>&1; do sleep 5; done' || exit 1
curl -X POST http://127.0.0.1:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL",
"messages": [{"role": "user", "content": "New York is a great city because..."}],
"max_tokens": 100
}'
Qwen3.6-35B-A3B is a thinking model, so with a small max_tokens budget the reply can finish while it is still reasoning. In that case message.content is empty and the generated text appears in message.reasoning_content. Increase max_tokens (Step 6 uses 2048) to get the final answer in content.
If you see curl: (7) Failed to connect, the server is still loading, the process exited (check the server log for OOM or path errors), or you are not curling the host that runs llama-server.
From another device, use http://<HARDWARE_IP>:30000/v1 where <HARDWARE_IP> is your hardware platform’s reachable address.
Example shape of the response for the request above (fields vary by llama.cpp version; message may include extra keys). The 100-token budget is spent on reasoning, so content is empty and finish_reason is length:
{
"choices": [
{
"finish_reason": "length",
"index": 0,
"message": {
"role": "assistant",
"content": "",
"reasoning_content": "Here's a thinking process:\n\n1. **Analyze User Input:** ..."
}
}
],
"created": 1765916539,
"model": "$MODEL_PATH",
"object": "chat.completion",
"usage": {
"completion_tokens": 100,
"prompt_tokens": 18,
"total_tokens": 118
},
"id": "chatcmpl-...",
"timings": {
...
}
}
Try a slightly longer prompt to confirm stable generation with Qwen3.6-35B-A3B:
curl -X POST http://127.0.0.1:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL",
"messages": [{"role": "user", "content": "Solve this step by step: If a train travels 120 miles in 2 hours, what is its average speed?"}],
"max_tokens": 2048
}'
Qwen3.6-35B-A3B may spend part of this token budget on reasoning before it writes the answer. If the response has an empty message.content and finish_reason is "length", increase max_tokens and try again.
Stop the server with Ctrl+C in the terminal where it is running.
Cleanup is optional rollback — not a required completion step.
WARNING
This permanently deletes the local llama.cpp build and the example model cache.
rm -rf ~/llama.cpp
rm -rf ~/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-MTP-GGUF
--ctx-size (or -c) to adjust for your needs. For agentic or coding workloads you need a minimum of 32768 tokens, preferably 100000 or more.--model to load any compatible GGUF downloaded locally; the llama.cpp server API stays the same. Use -hf to let llama.cpp manage downloads and updates. If you use --model with multi-modal models, provide a path to the .mmproj file with --mmproj. With -hf, the mmproj file loads automatically.http://<HARDWARE_IP>:30000/v1 using the OpenAI client pattern.The server implements the usual OpenAI-style chat features your llama.cpp build enables (including streaming and tool-related flows where supported).