Serve Models with llama.cpp

Symptom	Hardware platform	Cause	Fix
`cmake` fails with "CUDA not found"	All hardware platforms	CUDA toolkit not in PATH	Run `export PATH=/usr/local/cuda/bin:$PATH` and re-run CMake from a clean build directory
Build errors mentioning wrong GPU arch	All hardware platforms	CMake `CMAKE_CUDA_ARCHITECTURES` does not match the GPU	Use `-DCMAKE_CUDA_ARCHITECTURES=121a-real` as in the Instructions tab
GGUF download fails or stalls	All hardware platforms	Network or Hugging Face availability	Re-run the `-hf` launch from Step 4; it resumes partial files
"CUDA out of memory" when starting `llama-server`	All hardware platforms	Model too large for current context or available memory	Lower `--ctx-size` (e.g. 4096) or use a smaller quantization from the same repo
Server runs but latency is high	All hardware platforms	Layers not on GPU	Check `nvidia-smi` during a request to confirm the model is on the GPU; if not, set `--n-gpu-layers` explicitly as described in Step 4
`curl: (7) Failed to connect` on port 30000	All hardware platforms	No listener yet, wrong host, or crash	Wait for `server is listening`; run `curl` on the same host as `llama-server` (or use `<HARDWARE_IP>`); run `ss -tln` and confirm `:30000`; read server stderr for OOM or a bad `--model` path
Chat API errors or empty replies	All hardware platforms	Wrong `--model` path or incompatible GGUF	Verify the path to the `.gguf` file; update llama.cpp if the GGUF requires a newer format
Memory pressure within capacity	DGX Spark	UMA buffer cache not released	See UMA note below

NOTE

Unified memory (UMA). On hardware platforms with unified memory, GPU and CPU share memory dynamically. Some applications have not yet been updated for UMA, so you may hit memory issues even within capacity. If that happens, manually flush the buffer cache (use with care on shared systems):

Bash

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

For latest known issues, see the documentation linked under Resources for your hardware platform.