Serve LLMs with vLLM on One Device

Intermediate30 MIN

A locally accessible model API from one DGX Spark or Station

Overview Intermediate Advanced Troubleshooting

Set up a vLLM container as an NVIDIA Sync custom app

You will use NVIDIA Sync to connect to a DGX Spark or Station, then download a vLLM container and a model for your device.

Then, you will add a supplied launch script as a custom app in NVIDIA Sync. Once the app is set up, you can use Sync to start and stop the model and reach its API from your laptop.

Sync sets up the SSH tunnel and port forwarding for you.

Step 1
Install NVIDIA Sync on your laptop

First, download NVIDIA Sync for your laptop.

NVIDIA Sync is a desktop app that lets you connect to a remote Linux device and run apps on it.

Your OS has been auto-detected, but you can choose a different option to download.

Download

Then, install and open NVIDIA Sync.

For Windows: Double-click the .exe installer and follow the steps.

For macOS: Open nvidia-sync.dmg, drag the app into Applications, then open it.

For Ubuntu: Follow the Ubuntu install steps.

Success: NVIDIA Sync opens on your laptop.

Step 2
Add your DGX Spark or Station to NVIDIA Sync

First, choose how to add your device.

Your laptop must be able to reach the device on the network. You can add it in one of three ways:

mDNS: Select a device that Sync finds on the network, then enter your username and password
Manual: Select Add a device manually, enter the device's IP address, then enter your username and password
Import: Select a working SSH alias from your local SSH config, then add it

For mDNS or Manual, have these details ready:

Name: a name to help you find the device in Sync
Hostname or IP: the device's network name or IP address
Username: your username on the device
Password: the password for that user

NOTE

If you use an SSH key to sign in, first set up an SSH alias, then import it into Sync. See Add a device for a direct connection.

Then, allow NVIDIA Sync to set up the Docker group if prompted.

If Sync asks to add your user to the docker group, click Confirm. This lets the launch script run Docker as your user. You will need sudo access for this step.

NOTE

Access to the docker group grants root access without a password, which is a risk. Learn more in the Docker user guide.

Success: Your device appears in NVIDIA Sync.

Step 3
Connect, open a terminal and check the tools

First, connect to the device and open a terminal.

Select your device from the NVIDIA Sync drop-down
Click Connect
Select the Terminal app

The terminal opens an SSH session on the DGX device. You will run the download commands in this terminal.

Then, check that the Hugging Face CLI is installed.

Run:

Bash

hf --help

If it returns command not found, install the CLI with the Hugging Face install steps, then open a new terminal.

Success: hf --help shows the CLI help.

Step 4
Choose a model and download its files

First, choose one of the two rows for your device and memory size.

Each row has a matching launch script in Step 5. The recipe links give you more detail about the model settings used by these scripts.

Device	Memory	Model	Docker pull command	Model download command
DGX Spark	64GB unified memory	Qwen3.8 27B (NVFP4)	`docker pull vllm/vllm-openai:v0.28.0`	`hf download nvidia/Qwen3.8-27B-NVFP4`
DGX Spark	64GB unified memory	Nemotron 3.5 Bolt 74B A9B	`docker pull eugr/spark-vllm:nightly-20261016`	`hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB`
DGX Spark	128GB unified memory	Qwen3.8 Flash-Next	`docker pull eugr/spark-vllm-b12x:nightly-20261015`	`hf download nvidia/Qwen3.8-Flash-Next-FP4-Local`
DGX Spark	128GB unified memory	Qwen3.8 27B (NVFP4)	`docker pull nvcr.io/nvidia/vllm:26.10.pre1-py3`	`hf download nvidia/Qwen3.8-27B-NVFP4`
DGX Station	748GB coherent memory	Qwen3.8 Flash-Next	`docker pull vllm/vllm-openai:qwen38-flash-next`	`hf download Inferact/Qwen3.8-Flash-Next-NVFP4`
DGX Station	748GB coherent memory	Nemotron 3 Super 120B A12B	`docker pull vllm/vllm-openai:v0.31.0`	`hf download nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4`

Next, download the container and model on the DGX device.

Run the docker pull command from your row and wait for it to finish
Run the hf download command from the same row and wait for it to finish

NOTE

You can speed up the model download by temporarily setting HF_HUB_DISABLE_XET=1 in the download command, i.e. HF_HUB_DISABLE_XET=1 hf download <model>.

If you selected Qwen3.8 27B or Nemotron 3.5 Bolt, download the draft model needed for speculative decoding.

The Qwen3.8 27B and Nemotron 3.5 Bolt scripts use a second, smaller model to help speed up replies through something called "speculative decoding".

Run the command for your model below.

Model	Draft model download command
Qwen3.8 27B, on either Spark memory size	`hf download z-lab/Qwen3.8-27B-DFlash2`
Nemotron 3.5 Bolt 74B A9B	`hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr`

NOTE

The container and model files are large. Check that the device has enough free disk space and allow time for all downloads to finish.

Success: The container, model and any draft model have finished downloading.

Step 5
Add the model to NVIDIA Sync

First, open the launch script for the row you chose in Step 4.

Each script starts the container and runs the model with the recipe's settings. It also stops the container when you stop the app in Sync.

Device	Memory	App name	Launch script
DGX Spark	64GB	`vLLM-Qwen3.8-27B-64GB`	Qwen3.8 27B on 64GB Spark
DGX Spark	64GB	`vLLM-Nemotron3.5-Bolt`	Nemotron 3.5 Bolt on 64GB Spark
DGX Spark	128GB	`vLLM-Qwen3.8-Flash-Next`	Qwen3.8 Flash-Next on 128GB Spark
DGX Spark	128GB	`vLLM-Qwen3.8-27B-128GB`	Qwen3.8 27B on 128GB Spark
DGX Station	748GB	`vLLM-Qwen3.8-Flash-Next`	Qwen3.8 Flash-Next on Station
DGX Station	748GB	`vLLM-Nemotron3-Super`	Nemotron 3 Super on Station

Then, add the script as a custom app.

In NVIDIA Sync, connect to your device
Select Custom > + Add New
Enter the App name from your row in Name, and enter 8000 in Port
Paste the full script into Launch Script
Leave Auto open and Launch in Terminal off, and leave URL Path blank
Select Add

Learn more about Sync custom apps.

NOTE

Start with one model app at a time. If port 8000 is in use, choose a free port, such as 8001. Set both the Sync app's Port and PORT=8000 near the top of its launch script to that number. Use the same port in the laptop commands in Step 7.

Success: The app appears in the Custom section for your device.

Step 6
Launch the app and watch the server start

First, click the app name in NVIDIA Sync.

The script starts the container and loads the model. Sync forwards the model's API port to your laptop.

Next, open the resource monitor to watch the device.

Click the resource monitor in NVIDIA Sync
Select the views to see CPU, GPU and memory use as the server starts

The first start can take a while as vLLM builds runtime files and loads the model. Device activity shows that work is in progress; use Step 7 to check that the API is ready.

Then, check the container logs in a terminal on the DGX device.

Open Terminal through Sync and run the command for your app.

App name	Log command
`vLLM-Qwen3.8-27B-64GB`	`docker logs --tail 50 --follow vllm-spark-64gb-qwen38-27b`
`vLLM-Nemotron3.5-Bolt`	`docker logs --tail 50 --follow vllm-spark-64gb-nemotron35-bolt`
`vLLM-Qwen3.8-Flash-Next` (128GB DGX Spark)	`docker logs --tail 50 --follow vllm-spark-128gb-qwen38-flash-next`
`vLLM-Qwen3.8-27B-128GB`	`docker logs --tail 50 --follow vllm-spark-128gb-qwen38-27b`
`vLLM-Qwen3.8-Flash-Next` (DGX Station)	`docker logs --tail 50 --follow vllm-station-qwen38-flash-next`
`vLLM-Nemotron3-Super`	`docker logs --tail 50 --follow vllm-station-nemotron3-super`

Press Ctrl+C to leave the log view. The model app keeps running.

Success: The logs show the server is ready to accept requests. Continue to Step 7 to check the API and send a request.

Step 7
Test the API from your laptop

First, open a local terminal on your laptop and check the server.

Keep the model app running and Sync connected. Use the commands for your laptop below. If you chose a port other than 8000, use that port in each URL.

Run the health check and look for HTTP 200
Run the model-list command and copy the model ID from the reply

Laptop terminal	Health check	List served models
Windows PowerShell	`curl.exe --fail-with-body -i http://localhost:8000/health`	`curl.exe --fail-with-body -sS http://localhost:8000/v1/models`
macOS or Linux	`curl --fail-with-body -i http://localhost:8000/health`	`curl --fail-with-body -sS http://localhost:8000/v1/models`

If the health check cannot connect while the model loads, wait and try again. Check the container logs if startup fails.

Then, use the model ID to send a chat request.

Copy the chat command for your laptop
Replace <model-id> with the exact ID returned by /v1/models
Run the command and wait for the reply

Laptop terminal	Chat request
Windows PowerShell	`(Invoke-RestMethod -Uri http://localhost:8000/v1/chat/completions -Method Post -ContentType "application/json" -Body '{"model":"<model-id>","messages":[{"role":"user","content":"Write a haiku about a GPU."}],"max_tokens":4096}').choices`
macOS or Linux	`curl --fail-with-body -sS http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"<model-id>","messages":[{"role":"user","content":"Write a haiku about a GPU."}],"max_tokens":4096}'`

Success: The response has a choices array with the model's answer. Your laptop can now reach the model through Sync.

Step 8
Stop the model and start it again

First, stop the app through NVIDIA Sync.

Open the NVIDIA Sync pop-up window
Click the x beside the running custom app

The supplied script stops and removes its container. The model files and cached runtime files stay on the DGX device for the next start.

Then, start the model when you need it again.

Connect to the device through Sync
Click the custom app name
Wait for the model to load, then run the health check from Step 7

Success: The health check returns HTTP 200 again. You can start and stop this model through Sync each time you need it.

Next steps

Connect an app on your laptop to http://localhost:8000/v1, using the model ID from Step 7. Use your chosen port if you changed it.
Continue to the Advanced tab to learn how to choose and run a recipe yourself.
Use the Troubleshooting Tab if the app fails to start or your laptop cannot reach the API.
Use the vLLM cluster playbook to serve across more than one DGX Spark.

Resources

Serve LLMs with vLLM on One Device | RTX Workstation

Serve LLMs with vLLM on One Device

Set up a vLLM container as an NVIDIA Sync custom app

Step 1Install NVIDIA Sync on your laptop

Step 2Add your DGX Spark or Station to NVIDIA Sync

Step 3Connect, open a terminal and check the tools

Step 4Choose a model and download its files

Step 5Add the model to NVIDIA Sync

Step 6Launch the app and watch the server start

Step 7Test the API from your laptop

Step 8Stop the model and start it again

Next steps

Resources

Step 1
Install NVIDIA Sync on your laptop

Step 2
Add your DGX Spark or Station to NVIDIA Sync

Step 3
Connect, open a terminal and check the tools

Step 4
Choose a model and download its files

Step 5
Add the model to NVIDIA Sync

Step 6
Launch the app and watch the server start

Step 7
Test the API from your laptop

Step 8
Stop the model and start it again