---
title: "Serve LLMs with vLLM on One Device — Intermediate"
canonical: "https://build.nvidia.com/station/vllm/intermediate.md"
---

# Set up a vLLM container as an NVIDIA Sync custom app

You will use NVIDIA Sync to connect to a DGX Spark or Station, then download a vLLM container and a model for your device.

Then, you will add a supplied launch script as a custom app in NVIDIA Sync.
Once the app is set up, you can use Sync to start and stop the model and reach its API from your laptop.

Sync sets up the SSH tunnel and port forwarding for you.

## Step 1. Install NVIDIA Sync on your laptop

**First, download NVIDIA Sync for your laptop.**

NVIDIA Sync is a desktop app that lets you connect to a remote Linux device and run apps on it.

::spark-download

**Then, install and open NVIDIA Sync.**

**For Windows:** Double-click the `.exe` installer and follow the steps.

**For macOS:** Open `nvidia-sync.dmg`, drag the app into Applications, then open it.

**For Ubuntu:** Follow the [Ubuntu install steps](https://docs.nvidia.com/sync/latest/getting-started.html#installation).

**Success:** NVIDIA Sync opens on your laptop.

## Step 2. Add your DGX Spark or Station to NVIDIA Sync

**First, choose how to add your device.**

Your laptop must be able to reach the device on the network.
You can add it in one of three ways:

1. **mDNS:** Select a device that Sync finds on the network, then enter your username and password
2. **Manual:** Select **Add a device manually**, enter the device's IP address, then enter your username and password
3. **Import:** Select a working SSH alias from your local SSH config, then add it

For mDNS or Manual, have these details ready:

- **Name:** a name to help you find the device in Sync
- **Hostname or IP:** the device's network name or IP address
- **Username:** your username on the device
- **Password:** the password for that user

> [!NOTE]
> If you use an SSH key to sign in, first set up an SSH alias, then import it into Sync. See [Add a device for a direct connection](https://docs.nvidia.com/sync/latest/direct-connections.html#adding-a-device-for-a-direct-connection).

**Then, allow NVIDIA Sync to set up the Docker group if prompted.**

If Sync asks to add your user to the `docker` group, click **Confirm**.
This lets the launch script run Docker as your user.
You will need `sudo` access for this step.

> [!NOTE]
> Access to the `docker` group grants root access without a password, which is a risk. Learn more in the [Docker user guide](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user).

**Success:** Your device appears in NVIDIA Sync.

## Step 3. Connect, open a terminal and check the tools

**First, connect to the device and open a terminal.**

1. Select your device from the NVIDIA Sync drop-down
2. Click **Connect**
3. Select the **Terminal** app

The terminal opens an SSH session on the DGX device.
You will run the download commands in this terminal.

**Then, check that the Hugging Face CLI is installed.**

Run:

```bash
hf --help
```

If it returns `command not found`, install the CLI with the [Hugging Face install steps](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli#standalone-installer-recommended), then open a new terminal.

**Success:** `hf --help` shows the CLI help.

## Step 4. Choose a model and download its files

**First, choose one of the two rows for your device and memory size.**

Each row has a matching launch script in Step 5.
The recipe links give you more detail about the model settings used by these scripts.

| Device | Memory | Model | Docker pull command | Model download command |
| --- | --- | --- | --- | --- |
| DGX Spark | 64GB unified memory | [Qwen3.8 27B (NVFP4)](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark/index.html?model=qwen%2Fqwen3.8-27b&os=linux&backend=vllm&memory=64&precision=nvfp4&variant=&memoryConfig=64) | `docker pull vllm/vllm-openai:v0.28.0` | `hf download nvidia/Qwen3.8-27B-NVFP4` |
| DGX Spark | 64GB unified memory | [Nemotron 3.5 Bolt 74B A9B](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark/index.html?model=nvidia%2Fnemotron-3.5-bolt-74b-a9b&os=linux&backend=vllm&memory=64&precision=iq2&variant=&memoryConfig=64) | `docker pull eugr/spark-vllm:nightly-20261016` | `hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB` |
| DGX Spark | 128GB unified memory | [Qwen3.8 Flash-Next](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark/index.html?model=qwen%2Fqwen3.8-flash-next&os=linux&backend=vllm&memory=128&precision=fp4-local&variant=&memoryConfig=128) | `docker pull eugr/spark-vllm-b12x:nightly-20261015` | `hf download nvidia/Qwen3.8-Flash-Next-FP4-Local` |
| DGX Spark | 128GB unified memory | [Qwen3.8 27B (NVFP4)](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark/index.html?model=qwen%2Fqwen3.8-27b&os=linux&backend=vllm&memory=128&precision=nvfp4&variant=&memoryConfig=128) | `docker pull nvcr.io/nvidia/vllm:26.10.pre1-py3` | `hf download nvidia/Qwen3.8-27B-NVFP4` |
| DGX Station | 748GB coherent memory | [Qwen3.8 Flash-Next](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-station/index.html?model=qwen%2Fqwen3.8-flash-next&os=linux&backend=vllm&memory=&precision=nvfp4&variant=&memoryConfig=748) | `docker pull vllm/vllm-openai:qwen38-flash-next` | `hf download Inferact/Qwen3.8-Flash-Next-NVFP4` |
| DGX Station | 748GB coherent memory | [Nemotron 3 Super 120B A12B](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-station/index.html?model=nvidia%2Fnemotron-3-super-120b-a12b&os=linux&backend=vllm&memory=&precision=nvfp4&variant=mtp&memoryConfig=748) | `docker pull vllm/vllm-openai:v0.31.0` | `hf download nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4` |

**Next, download the container and model on the DGX device.**

1. Run the `docker pull` command from your row and wait for it to finish
2. Run the `hf download` command from the same row and wait for it to finish

> [!NOTE]
> You can speed up the model download by temporarily setting `HF_HUB_DISABLE_XET=1` in the download command, i.e. `HF_HUB_DISABLE_XET=1 hf download <model>`. 

**If you selected Qwen3.8 27B or Nemotron 3.5 Bolt, download the draft model needed for speculative decoding.**

The Qwen3.8 27B and Nemotron 3.5 Bolt scripts use a second, smaller model to help speed up replies through something called "[speculative decoding](https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/)".

Run the command for your model below.

| Model | Draft model download command |
| --- | --- |
| Qwen3.8 27B, on either Spark memory size | `hf download z-lab/Qwen3.8-27B-DFlash2` |
| Nemotron 3.5 Bolt 74B A9B | `hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr` |

> [!NOTE]
> The container and model files are large. Check that the device has enough free disk space and allow time for all downloads to finish.

**Success:** The container, model and any draft model have finished downloading.

## Step 5. Add the model to NVIDIA Sync

**First, open the launch script for the row you chose in Step 4.**

Each script starts the container and runs the model with the recipe's settings. It also stops the container when you stop the app in Sync.

| Device | Memory | App name | Launch script |
| --- | --- | --- | --- |
| DGX Spark | 64GB | `vLLM-Qwen3.8-27B-64GB` | [Qwen3.8 27B on 64GB Spark](https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-vllm/assets/sync-vllm-spark-64gb-qwen38-27b.sh) |
| DGX Spark | 64GB | `vLLM-Nemotron3.5-Bolt` | [Nemotron 3.5 Bolt on 64GB Spark](https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-vllm/assets/sync-vllm-spark-64gb-nemotron35-bolt.sh) |
| DGX Spark | 128GB | `vLLM-Qwen3.8-Flash-Next` | [Qwen3.8 Flash-Next on 128GB Spark](https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-vllm/assets/sync-vllm-spark-128gb-qwen38-flash-next.sh) |
| DGX Spark | 128GB | `vLLM-Qwen3.8-27B-128GB` | [Qwen3.8 27B on 128GB Spark](https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-vllm/assets/sync-vllm-spark-128gb-qwen38-27b.sh) |
| DGX Station | 748GB | `vLLM-Qwen3.8-Flash-Next` | [Qwen3.8 Flash-Next on Station](https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-vllm/assets/sync-vllm-station-qwen38-flash-next.sh) |
| DGX Station | 748GB | `vLLM-Nemotron3-Super` | [Nemotron 3 Super on Station](https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-vllm/assets/sync-vllm-station-nemotron3-super.sh) |

**Then, add the script as a custom app.**

1. In NVIDIA Sync, connect to your device
2. Select **Custom** > **+ Add New**
3. Enter the **App name** from your row in **Name**, and enter `8000` in **Port**
4. Paste the full script into **Launch Script**
5. Leave **Auto open** and **Launch in Terminal** off, and leave **URL Path** blank
6. Select **Add**

Learn more about [Sync custom apps](https://docs.nvidia.com/sync/latest/applications.html#custom-scripts-and-applications).

> [!NOTE]
> Start with one model app at a time. If port `8000` is in use, choose a free port, such as `8001`. Set both the Sync app's **Port** and `PORT=8000` near the top of its launch script to that number. Use the same port in the laptop commands in Step 7.

**Success:** The app appears in the **Custom** section for your device.

## Step 6. Launch the app and watch the server start

**First, click the app name in NVIDIA Sync.**

The script starts the container and loads the model. Sync forwards the model's API port to your laptop.

**Next, open the resource monitor to watch the device.**

1. Click the resource monitor in NVIDIA Sync
2. Select the views to see CPU, GPU and memory use as the server starts

The first start can take a while as vLLM builds runtime files and loads the model. Device activity shows that work is in progress; use Step 7 to check that the API is ready.

**Then, check the container logs in a terminal on the DGX device.**

Open **Terminal** through Sync and run the command for your app.

| App name | Log command |
| --- | --- |
| `vLLM-Qwen3.8-27B-64GB` | `docker logs --tail 50 --follow vllm-spark-64gb-qwen38-27b` |
| `vLLM-Nemotron3.5-Bolt` | `docker logs --tail 50 --follow vllm-spark-64gb-nemotron35-bolt` |
| `vLLM-Qwen3.8-Flash-Next` (128GB DGX Spark) | `docker logs --tail 50 --follow vllm-spark-128gb-qwen38-flash-next` |
| `vLLM-Qwen3.8-27B-128GB` | `docker logs --tail 50 --follow vllm-spark-128gb-qwen38-27b` |
| `vLLM-Qwen3.8-Flash-Next` (DGX Station) | `docker logs --tail 50 --follow vllm-station-qwen38-flash-next` |
| `vLLM-Nemotron3-Super` | `docker logs --tail 50 --follow vllm-station-nemotron3-super` |

Press `Ctrl+C` to leave the log view. The model app keeps running.

**Success:** The logs show the server is ready to accept requests. Continue to Step 7 to check the API and send a request.

## Step 7. Test the API from your laptop

**First, open a local terminal on your laptop and check the server.**

Keep the model app running and Sync connected.
Use the commands for your laptop below.
If you chose a port other than `8000`, use that port in each URL.

1. Run the health check and look for HTTP `200`
2. Run the model-list command and copy the model ID from the reply

| Laptop terminal | Health check | List served models |
| --- | --- | --- |
| Windows PowerShell | `curl.exe --fail-with-body -i http://localhost:8000/health` | `curl.exe --fail-with-body -sS http://localhost:8000/v1/models` |
| macOS or Linux | `curl --fail-with-body -i http://localhost:8000/health` | `curl --fail-with-body -sS http://localhost:8000/v1/models` |

If the health check cannot connect while the model loads, wait and try again. Check the container logs if startup fails.

**Then, use the model ID to send a chat request.**

1. Copy the chat command for your laptop
2. Replace `<model-id>` with the exact ID returned by `/v1/models`
3. Run the command and wait for the reply

| Laptop terminal | Chat request |
| --- | --- |
| Windows PowerShell | `(Invoke-RestMethod -Uri http://localhost:8000/v1/chat/completions -Method Post -ContentType "application/json" -Body '{"model":"<model-id>","messages":[{"role":"user","content":"Write a haiku about a GPU."}],"max_tokens":4096}').choices` |
| macOS or Linux | `curl --fail-with-body -sS http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"<model-id>","messages":[{"role":"user","content":"Write a haiku about a GPU."}],"max_tokens":4096}'` |

**Success:** The response has a `choices` array with the model's answer. Your laptop can now reach the model through Sync.

## Step 8. Stop the model and start it again

**First, stop the app through NVIDIA Sync.**

1. Open the NVIDIA Sync pop-up window
2. Click the **x** beside the running custom app

The supplied script stops and removes its container. The model files and cached runtime files stay on the DGX device for the next start.

**Then, start the model when you need it again.**

1. Connect to the device through Sync
2. Click the custom app name
3. Wait for the model to load, then run the health check from Step 7

**Success:** The health check returns HTTP `200` again. You can start and stop this model through Sync each time you need it.

## Next steps

- Connect an app on your laptop to `http://localhost:8000/v1`, using the model ID from Step 7. Use your chosen port if you changed it.
- Continue to the Advanced tab to learn how to choose and run a recipe yourself.
- Use the Troubleshooting Tab if the app fails to start or your laptop cannot reach the API.
- Use [the vLLM cluster playbook](https://build.nvidia.com/playbooks/vllm-cluster) to serve across more than one DGX Spark.