---
title: "Run LLMs on Your NVIDIA Device — Guide"
canonical: "https://build.nvidia.com/playbooks/running-llamacpp/overview.md"
---

# Basic Idea

Choose a model and spin it up locally on your NVIDIA system for privacy, control, and unlimited free experimentation without any token anxiety.

Most users will not require this playbook. Apps that use local AI will often configure a model for you. Follow this playbook when you want to download and run a local model yourself.

# Step 1: Choose Your Model

The [NVIDIA Local AI Inference Recipes for RTX Spark](https://nvidia.github.io/local-ai-inference-recipes/hardware/rtx-spark/index.html) currently recommends **Qwen3.8-27B**. This playbook uses that model throughout.

Here are the recommended models based on the available VRAM on your system:

| Target VRAM | Recommended LLM |
| --- | --- |
| 8–16 GB | Qwen 3.5 9B |
| 16–24 GB | Qwen 3.6 35B |
| 24–48 GB | Qwen3.8 27B |
| 48–128 GB | Qwen3.8 27B |
| 128–256 GB | Qwen3.8-Flash-Next |
| 256–384 GB | DeepSeek V4 Flash 0731 |
| 384–500 GB | GLM5.3 Flash |
| 500 GB+ | GLM 5.3 |

# Step 2: Choose Your Backend

- **llama.cpp (Recommended)** — Fully control how models run on your hardware for maximum performance. Choose it when building an app that needs more control over speed, memory use, and hardware.
- **LM Studio Bionic** — Try models through an easy-to-use desktop app. Choose Bionic to download local models, chat with them, and test ideas with little setup.
- **Ollama** — Easily connect local models to your apps and agents. Choose it to run and manage models with simple commands.

## Option 1: llama.cpp (Recommended)

### Download

- GGML, the maintainers of the llama.cpp project, provide daily builds of the llama.cpp project for RTX Spark. To get the latest - navigate to the [llama.cpp releases page](https://github.com/ggml-org/llama.cpp/releases) and download the links for Windows arm64 (CUDA 13) and CUDA 13.4 DLLs.
- Create a dedicated folder for llama.cpp, such as `C:\llama.cpp`.
- Extract the contents of the Windows arm64 (CUDA 13) zip into that folder.
- Extract the contents of the CUDA 13.4 DLLs zip into the **same** folder. Do not create a second nested folder. If Windows asks whether to merge folders, allow it.
- Confirm that `llama-server.exe`, `llama-cli.exe`, and the CUDA runtime files, including `cudart64_*.dll`, are directly inside `C:\llama.cpp`. Windows must be able to find the CUDA DLLs beside the llama.cpp executables.

### Run

- Find the model recipe:
- Open [NVIDIA Local AI Inference Recipes for RTX Spark](https://nvidia.github.io/local-ai-inference-recipes/hardware/rtx-spark/index.html). Select **RTX Spark**, **Windows**, **Qwen3.8-27B**, and **llama.cpp**.
- Check the prerequisites and memory requirements. The command below matches the current recommended Q4_K_M Windows recipe.
- Run the model:
- Open the folder containing `llama-server.exe` in File Explorer. Click the address bar, type `powershell`, and press Enter.
- Run the following command. The `.\` prefix tells PowerShell to use `llama-server.exe` from the current folder.

```powershell
.\llama-server.exe -hf "unsloth/Qwen3.8-27B-GGUF" `
-hff "Qwen3.8-27B-UD-Q4_K_M.gguf" `
--mmproj-url "https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/4ca720788d1e01f1bff70c033e0d0028fd02e502/mmproj-F16.gguf" `
--host 127.0.0.1 `
--port 9931 `
--batch-size 4096 `
--ubatch-size 512 `
--ctx-size 262144 `
--parallel 1 `
-ngl 999 `
--spec-type draft-dflash `
--spec-draft-n-max 7 `
-hfrd "incoai/Qwen3.8-27B-DFlash2-GGUF" `
--spec-draft-model "Qwen3.8-27B-DFlash2-Q4_K_M.gguf" `
--spec-draft-backend-sampling `
--temp 1.0 `
--top-p 0.95 `
--top-k 20 `
--min-p 0.0 `
--presence-penalty 0.0 `
--repeat-penalty 1.0
```

- Press Enter. Recipes using Hugging Face download options automatically download and cache the model files on first launch. Wait for downloading and loading to finish, and keep PowerShell open while using the model.

During DFlash startup, llama.cpp may print a red line that begins `failed to initialize the context: dflash requires ctx_other to be set`, followed by `failed to measure the memory of the extra model, fitting without it`. The first line says the warning is normal during memory fitting. These messages are an expected memory-fitting probe, not a failed launch. Continue waiting until the server reports that it is listening on `http://127.0.0.1:9931`. If the process exits or never reports that it is listening, the server did not finish starting.

### Test

After launch, navigate to [http://127.0.0.1:9931](http://127.0.0.1:9931) in your preferred browser and send a message in the web UI.

Services that support OpenAI-compatible APIs can use the base URL `http://127.0.0.1:9931/v1`. To verify that endpoint directly, open a second PowerShell window and run:

```powershell
'{"model":"unsloth/Qwen3.8-27B-GGUF","messages":[{"role":"user","content":"Hello"}]}' | curl.exe 'http://127.0.0.1:9931/v1/chat/completions' -H 'Content-Type: application/json' --data-binary '@-'
```

The response should contain a reply from the model.

## Option 2: LM Studio Bionic

### Download

Download and run the [LM Studio Bionic installer for Windows](https://lmstudio.ai/download/bionic/latest/win32/x64?redirectVersion=2). Bionic is a separate app from classic LM Studio; use the Bionic installer for this workflow.

### Run

- Open Bionic, select **Choose a model**, and then select **Get local models** to browse models that can run on your device.

![LM Studio Bionic model picker with Get local models highlighted](https://assets.ngc.nvidia.com/products/api-catalog/rtx-spark/lm-studio-bionic-get-local-models.png)

- On the Explore screen, select **Qwen3.8 27B**. Confirm that the download option shows the recommended `GGUF` `Q4_K_M` variant, and then select **Download**.

![LM Studio Bionic Qwen3.8 27B download button highlighted](https://assets.ngc.nvidia.com/products/api-catalog/rtx-spark/lm-studio-bionic-download-qwen38-27b.png)

- When the download completes, return to your Bionic session and choose the downloaded Qwen3.8 27B model from the model picker.

### Test

Enter a message in the Bionic session. A response confirms that the local model is ready to use.

## Option 3: Ollama

### Download

- Open a Powershell window on your machine and execute:

```powershell
irm https://ollama.com/install.ps1 | iex
```

After installation, Ollama starts automatically in the background and serves its local API on `http://localhost:11434`. You do not need to run `ollama serve` or reopen the app.

### Run

Download the recommended Qwen3.8-27B model:

```powershell
ollama pull qwen3.8:27b
```

When the download completes, `ollama ls` should list `qwen3.8:27b`. The Ollama tag already packages the Q4_K_M model and its Ollama parameters. You do not need to copy the llama.cpp launch recipe or choose a separate quantization. Other models are available in the [Ollama model library](https://ollama.com/library), but this playbook uses Qwen3.8-27B consistently across all three backends.

### Test

Start an interactive chat with the downloaded model:

```powershell
ollama run qwen3.8:27b
```

Type your prompt or question and press Enter.

# Next Steps and Cleanup

Downloaded models remain on disk so you can reuse them without downloading them again. When you finish, stop only the backend you started:

| Backend | Stop the model or service | Process and port | Downloaded model files remain in |
| --- | --- | --- | --- |
| llama.cpp | Press `Ctrl+C` in the PowerShell window running the server. If that window is unresponsive, end `llama-server.exe` in Task Manager. | `llama-server.exe` on `127.0.0.1:9931` | `%USERPROFILE%\.cache\huggingface\hub`, unless you configured a different Hugging Face or llama.cpp cache location |
| LM Studio Bionic | Close the Bionic app. | LM Studio Bionic desktop process; no API port is opened by the steps in this playbook | Bionic's local model library; open **Settings > Local Models > Library** to manage downloaded models |
| Ollama | Run `ollama stop qwen3.8:27b` to unload the model. To stop the background service, quit Ollama from its taskbar icon. | Ollama background app on `127.0.0.1:11434` | `%USERPROFILE%\.ollama\models`, unless `OLLAMA_MODELS` points elsewhere |

Stopping a backend frees the memory and port it was using; it does not delete the downloaded model. Delete a model from the backend's model library or cache only if you want to reclaim disk space.