Serve Models with llama.cpp

Agent-ready models

Agent-ready models are tuned for agentic workloads — tool calling, reasoning traces, and long multi-turn sessions. Use this tab to pick a recommended model for your hardware platform and follow the launch guidance below.

Complete Steps 1–3 in the Instructions tab first (dependencies, clone, and CUDA build). Then launch from your ~/llama.cpp directory with the recipe below.

Recommendations by hardware platform

DGX Spark: Agent-ready Qwen3.6-35B-A3B (GGUF, MTP)

Agentic serving for the recommended model on unified-memory hardware. MTP speculative decoding and preserve_thinking keep interleaved thinking blocks in history for multi-turn agent sessions.

From ~/llama.cpp:

./build/bin/llama-server \
  -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \
  --host 0.0.0.0 \
  --port 30000 \
  --chat-template-kwargs '{"preserve_thinking": true}' \
  --spec-type draft-mtp \
  --spec-draft-n-max 3

Settings used:

• -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL — MTP-compatible GGUF with a quality/speed balance for unified memory
• --chat-template-kwargs '{"preserve_thinking": true}' — preserves prior thinking blocks for interleaved thinking in agentic workflows
• --spec-type draft-mtp / --spec-draft-n-max 3 — Multi-Token Prediction speculative decoding for faster token generation
• Context: for agentic or coding needs, prefer at least 32768 tokens (preferably 100000+); override with --ctx-size / -c if needed

Verify your server

After you launch the recipe above, confirm startup using Step 5. Test the API in the Instructions tab.

Next steps

• General serving workflow: build, health checks, and API testing — see the Instructions tab
• Other GGUF models: see Find model recipes in the Overview tab