Run LLMs on Your NVIDIA Device

Basic Idea

Choose a model and spin it up locally on your NVIDIA system for privacy, control, and unlimited free experimentation without any token anxiety.

Most users will not require this playbook. Apps that use local AI will often configure a model for you. Follow this playbook when you want to download and run a local model yourself.

Choose Your Model

The NVIDIA Local AI Inference Recipes for RTX Spark currently recommends Qwen3.8-27B. This playbook uses that model throughout.

Here are the recommended models based on the available VRAM on your system:

Choose Your Backend

• llama.cpp (Recommended) — Fully control how models run on your hardware for maximum performance. Choose it when building an app that needs more control over speed, memory use, and hardware.
• LM Studio Bionic — Try models through an easy-to-use desktop app. Choose Bionic to download local models, chat with them, and test ideas with little setup.
• Ollama — Easily connect local models to your apps and agents. Choose it to run and manage models with simple commands.

Option 1: llama.cpp (Recommended)

Download

• GGML, the maintainers of the llama.cpp project, provide daily builds of the llama.cpp project for RTX Spark. To get the latest - navigate to the llama.cpp releases page and download the links for Windows arm64 (CUDA 13) and CUDA 13.4 DLLs.
• Create a dedicated folder for llama.cpp, such as C:\llama.cpp.
• Extract the contents of the Windows arm64 (CUDA 13) zip into that folder.
• Extract the contents of the CUDA 13.4 DLLs zip into the same folder. Do not create a second nested folder. If Windows asks whether to merge folders, allow it.
• Confirm that llama-server.exe, llama-cli.exe, and the CUDA runtime files, including cudart64_*.dll, are directly inside C:\llama.cpp. Windows must be able to find the CUDA DLLs beside the llama.cpp executables.

Run

• Find the model recipe:
  • Open NVIDIA Local AI Inference Recipes for RTX Spark. Select RTX Spark, Windows, Qwen3.8-27B, and llama.cpp.
  • Check the prerequisites and memory requirements. The command below matches the current recommended Q4_K_M Windows recipe.
• Run the model:
  • Open the folder containing llama-server.exe in File Explorer. Click the address bar, type powershell, and press Enter.
  • Run the following command. The .\ prefix tells PowerShell to use llama-server.exe from the current folder.

.\llama-server.exe -hf "unsloth/Qwen3.8-27B-GGUF" `
  -hff "Qwen3.8-27B-UD-Q4_K_M.gguf" `
  --mmproj-url "https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/4ca720788d1e01f1bff70c033e0d0028fd02e502/mmproj-F16.gguf" `
  --host 127.0.0.1 `
  --port 9931 `
  --batch-size 4096 `
  --ubatch-size 512 `
  --ctx-size 262144 `
  --parallel 1 `
  -ngl 999 `
  --spec-type draft-dflash `
  --spec-draft-n-max 7 `
  -hfrd "incoai/Qwen3.8-27B-DFlash2-GGUF" `
  --spec-draft-model "Qwen3.8-27B-DFlash2-Q4_K_M.gguf" `
  --spec-draft-backend-sampling `
  --temp 1.0 `
  --top-p 0.95 `
  --top-k 20 `
  --min-p 0.0 `
  --presence-penalty 0.0 `
  --repeat-penalty 1.0

• Press Enter. Recipes using Hugging Face download options automatically download and cache the model files on first launch. Wait for downloading and loading to finish, and keep PowerShell open while using the model.

During DFlash startup, llama.cpp may print a red line that begins failed to initialize the context: dflash requires ctx_other to be set, followed by failed to measure the memory of the extra model, fitting without it. The first line says the warning is normal during memory fitting. These messages are an expected memory-fitting probe, not a failed launch. Continue waiting until the server reports that it is listening on http://127.0.0.1:9931. If the process exits or never reports that it is listening, the server did not finish starting.

Test

After launch, navigate to http://127.0.0.1:9931 in your preferred browser and send a message in the web UI.

Services that support OpenAI-compatible APIs can use the base URL http://127.0.0.1:9931/v1. To verify that endpoint directly, open a second PowerShell window and run:

'{"model":"unsloth/Qwen3.8-27B-GGUF","messages":[{"role":"user","content":"Hello"}]}' | curl.exe 'http://127.0.0.1:9931/v1/chat/completions' -H 'Content-Type: application/json' --data-binary '@-'

The response should contain a reply from the model.

Option 2: LM Studio Bionic

Download

Download and run the LM Studio Bionic installer for Windows. Bionic is a separate app from classic LM Studio; use the Bionic installer for this workflow.

Run

• Open Bionic, select Choose a model, and then select Get local models to browse models that can run on your device.

• On the Explore screen, select Qwen3.8 27B. Confirm that the download option shows the recommended GGUF Q4_K_M variant, and then select Download.

• When the download completes, return to your Bionic session and choose the downloaded Qwen3.8 27B model from the model picker.

Test

Enter a message in the Bionic session. A response confirms that the local model is ready to use.

Option 3: Ollama

Download

• Open a Powershell window on your machine and execute:

irm https://ollama.com/install.ps1 | iex

After installation, Ollama starts automatically in the background and serves its local API on http://localhost:11434. You do not need to run ollama serve or reopen the app.

Run

Download the recommended Qwen3.8-27B model:

ollama pull qwen3.8:27b

When the download completes, ollama ls should list qwen3.8:27b. The Ollama tag already packages the Q4_K_M model and its Ollama parameters. You do not need to copy the llama.cpp launch recipe or choose a separate quantization. Other models are available in the Ollama model library, but this playbook uses Qwen3.8-27B consistently across all three backends.

Test

Start an interactive chat with the downloaded model:

ollama run qwen3.8:27b

Type your prompt or question and press Enter.

Next Steps and Cleanup

Downloaded models remain on disk so you can reuse them without downloading them again. When you finish, stop only the backend you started:

Stopping a backend frees the memory and port it was using; it does not delete the downloaded model. Delete a model from the backend's model library or cache only if you want to reclaim disk space.