--- title: "Serve Models with llama.cpp — Instructions" canonical: "https://build.nvidia.com/playbooks/llama-cpp/instructions.md" --- # Install with an agent ```text Use https://build.nvidia.com/playbooks/llama-cpp.md and complete this playbook on this machine: Serve Models with llama.cpp. Fetch that .md URL directly; the non-.md page is a JavaScript shell. When finished, state clearly whether the playbook succeeded, and give the evidence: what you installed, where, and the output of a command proving it works (a version string, a container listing, or an HTTP status). ``` > [!NOTE] > This has been tested using OpenCode and Qwen 3.8 35B A3B NVFP4. Performance and outcome may vary based on agent and model selection. # Step 1. Install the dependencies Update the package metadata and install the required dependencies: ```bash sudo apt update sudo apt install -y git clang cmake libcurl4-openssl-dev libssl-dev ``` # Step 2. Clone the llama.cpp repository Clone upstream llama.cpp — the framework you are building: ```bash git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp cd ~/llama.cpp ``` # Step 3. Build llama.cpp with CUDA Configure CMake with CUDA so GGML’s CUDA backend matches your hardware platform. For the platforms in this playbook, use CUDA architecture `121a-real`: ```bash cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CURL=ON -DGGML_RPC=ON -DCMAKE_CUDA_ARCHITECTURES=121a-real cmake --build build --config Release --target llama-server -j ``` The build usually takes on the order of 5–10 minutes. When it finishes, `llama-server` appears under `build/bin/`. # Step 4. Start llama-server with a model llama.cpp loads models in **GGUF** format. This playbook uses the **Q4_K_XL** checkpoint from `unsloth/Qwen3.6-35B-A3B-MTP-GGUF`, which provides a good balance between quality and speed on the supported hardware platforms. From the `~/llama.cpp` directory you cloned in Step 2, launch the OpenAI-compatible server with GPU offload. It loads the model from Hugging Face first if it has not been downloaded before or if there are updates. All models are saved in the default Hugging Face cache directory under `~/.cache/huggingface/hub`. For example, this model is saved into `~/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-MTP-GGUF`. It also automatically loads an mmproj file to enable vision capabilities if supported by the model. By default, `llama-server` tries to fit the full model context with the ability to serve 4 concurrent requests, and adjusts parameters automatically if needed. ```bash ./build/bin/llama-server \ -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \ --host 0.0.0.0 \ --port 30000 ``` To run with MTP speculative decoding, provide additional parameters as shown below. MTP requires a compatible model, like `unsloth/Qwen3.6-35B-A3B-MTP-GGUF` used in this example. The following example also sets the `preserve_thinking` flag so Qwen models can use interleaved thinking by preserving prior thinking blocks in the history — useful for agentic workflows. For a dedicated agentic launch path, see the **Agent-ready Models** tab. ```bash ./build/bin/llama-server \ -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \ --host 0.0.0.0 \ --port 30000 \ --chat-template-kwargs '{"preserve_thinking": true}' \ --spec-type draft-mtp \ --spec-draft-n-max 3 ``` **Parameters (short):** - `--host` / `--port`: bind address and port for the HTTP API - `--n-gpu-layers`: number of model layers to offload to the GPU; llama.cpp normally chooses this for you, so set it explicitly (for example `--n-gpu-layers 99`) only if `nvidia-smi` shows the model is not running on the GPU - `--chat-template-kwargs`: additional params for the JSON template parser; must be a valid JSON object string - `--spec-type`: comma-separated list of speculative decoding types (default: none; most MTP-compatible models use `draft-mtp` — check the model card) - `--spec-draft-n-max`: number of tokens to draft for speculative decoding (default: 3) You should see log lines similar to: ``` 0.14.322.968 I srv load_model: speculative decoding context initialized 0.14.322.970 I slot load_model: id 0 | task -1 | new slot, n_ctx = 262144 0.14.322.972 I slot load_model: id 1 | task -1 | new slot, n_ctx = 262144 0.14.322.972 I slot load_model: id 2 | task -1 | new slot, n_ctx = 262144 0.14.322.973 I slot load_model: id 3 | task -1 | new slot, n_ctx = 262144 0.14.323.063 I srv load_model: prompt cache is enabled, size limit: 8192 MiB ... 0.14.342.935 I srv llama_server: model loaded 0.14.342.939 I srv llama_server: server is listening on http://0.0.0.0:30000 0.14.342.944 I srv update_slots: all slots are idle ``` **Keep this terminal open** while testing. Large GGUFs can take a minute or more to load, and the initial model download can take a while if the model is not downloaded yet. You will see a progress bar when the model is being downloaded. The server is ready to accept connections on port 30000 after you see the `server is listening` message (see Troubleshooting if `curl` reports connection refused). # Step 5. Test the API Before sending requests, confirm the server is ready (large models can take several minutes to load): ```bash timeout 900 bash -c 'until curl -sf http://127.0.0.1:30000/health > /dev/null 2>&1; do sleep 5; done' || exit 1 ``` ```bash curl -X POST http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL", "messages": [{"role": "user", "content": "New York is a great city because..."}], "max_tokens": 100 }' ``` Qwen3.6-35B-A3B is a thinking model, so with a small `max_tokens` budget the reply can finish while it is still reasoning. In that case `message.content` is empty and the generated text appears in `message.reasoning_content`. Increase `max_tokens` (Step 6 uses 2048) to get the final answer in `content`. If you see `curl: (7) Failed to connect`, the server is still loading, the process exited (check the server log for OOM or path errors), or you are not curling the host that runs `llama-server`. From another device, use `http://:30000/v1` where `` is your hardware platform’s reachable address. Example shape of the response for the request above (fields vary by llama.cpp version; `message` may include extra keys). The 100-token budget is spent on reasoning, so `content` is empty and `finish_reason` is `length`: ```json { "choices": [ { "finish_reason": "length", "index": 0, "message": { "role": "assistant", "content": "", "reasoning_content": "Here's a thinking process:\n\n1. **Analyze User Input:** ..." } } ], "created": 1765916539, "model": "$MODEL_PATH", "object": "chat.completion", "usage": { "completion_tokens": 100, "prompt_tokens": 18, "total_tokens": 118 }, "id": "chatcmpl-...", "timings": { ... } } ``` # Step 6. Longer completion (with Qwen3.6-35B-A3B) Try a slightly longer prompt to confirm stable generation with **Qwen3.6-35B-A3B**: ```bash curl -X POST http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL", "messages": [{"role": "user", "content": "Solve this step by step: If a train travels 120 miles in 2 hours, what is its average speed?"}], "max_tokens": 2048 }' ``` Qwen3.6-35B-A3B may spend part of this token budget on reasoning before it writes the answer. If the response has an empty `message.content` and `finish_reason` is `"length"`, increase `max_tokens` and try again. # Step 7. Cleanup Stop the server with `Ctrl+C` in the terminal where it is running. Cleanup is optional rollback — not a required completion step. > [!WARNING] > This permanently deletes the local llama.cpp build and the example model cache. ```bash rm -rf ~/llama.cpp rm -rf ~/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-MTP-GGUF ``` # Step 8. Next steps 1. **Context length:** By default, llama.cpp tries to allocate the maximum context size supported for the model if possible. Set it manually with `--ctx-size` (or `-c`) to adjust for your needs. For agentic or coding workloads you need a minimum of 32768 tokens, preferably 100000 or more. 2. **Other models:** Use `--model` to load any compatible GGUF downloaded locally; the llama.cpp server API stays the same. Use `-hf` to let llama.cpp manage downloads and updates. If you use `--model` with multi-modal models, provide a path to the `.mmproj` file with `--mmproj`. With `-hf`, the mmproj file loads automatically. 3. **Integrations:** Point Open WebUI, Continue.dev, or custom clients at `http://:30000/v1` using the OpenAI client pattern. The server implements the usual OpenAI-style chat features your llama.cpp build enables (including streaming and tool-related flows where supported).