Chat with LLMs Using Open WebUI and Ollama

Use NVIDIA Sync to connect and open a terminal

From your laptop:

• Open NVIDIA Sync with the desktop icon or from the system tray or taskbar.
• Select your hardware platform from the device dropdown.
• Select Connect.
• After the connection is established, select Terminal to open a terminal on the hardware platform.

Configure Docker permissions

You must first make sure that your user account can run Docker commands on the hardware platform without sudo.

To test that, in the terminal run:

docker ps > /dev/null

Success: If the command returns a blank, then skip ahead to Step 3.

Otherwise, you will see a permission denied error, which means you still need to remove the sudo requirement. To do that, add your user to the docker group with the commands below.

sudo usermod -aG docker $USER
newgrp docker

Then verify the change is set by testing Docker access again with the command:

docker ps > /dev/null

Download the Open WebUI container image

Pull the container image onto your hardware platform with the command:

docker pull ghcr.io/open-webui/open-webui:ollama

Wait for the image to download, then go to Step 4.

Add Open WebUI as a custom application through NVIDIA Sync

A custom application lets NVIDIA Sync start Open WebUI and automatically forward its port.

In the NVIDIA Sync device window:

• Select Add New in the Custom section.

• Fill out the form with these values:

  • Name: Open WebUI
  • Port: 12000
  • Auto open in browser at the following path: Check this checkbox

• Then, copy and paste the entire script below into the Launch Script field

#!/usr/bin/env bash
set -euo pipefail

NAME="open-webui"
IMAGE="ghcr.io/open-webui/open-webui:ollama"

cleanup() {
  echo "Signal received; stopping ${NAME}..."
  docker stop "${NAME}" >/dev/null 2>&1 || true
  exit 0
}
trap cleanup INT TERM HUP QUIT EXIT

# Ensure Docker CLI and daemon are available
if ! docker info >/dev/null 2>&1; then
  echo "Error: Docker daemon not reachable." >&2
  exit 1
fi

# Already running?
if [ -n "$(docker ps -q --filter "name=^${NAME}$" --filter "status=running")" ]; then
  echo "Container ${NAME} is already running."
else
  # Exists but stopped? Start it.
  if [ -n "$(docker ps -aq --filter "name=^${NAME}$")" ]; then
    echo "Starting existing container ${NAME}..."
    docker start "${NAME}" >/dev/null
  else
    # Not present: create and start it.
    echo "Creating and starting ${NAME}..."
    docker run -d -p 12000:8080 --gpus=all \
      -v open-webui:/app/backend/data \
      -v open-webui-ollama:/root/.ollama \
      --name "${NAME}" "${IMAGE}" >/dev/null
  fi
fi

echo "Running. Press Ctrl+C to stop ${NAME}."
# Keep the script alive until a signal arrives
while :; do sleep 86400; done

• Finally, click the Add button to finish the configuration.

Launch Open WebUI and create an administrator account

Once the app is configured, you can launch it from NVIDIA Sync and connect to it with your browser.

In the NVIDIA Sync application window for your hardware platform, select Open WebUI in the Custom section.

The application should open in your web browser at the URL http://localhost:12000.

If it does not, open your web browser and go to http://localhost:12000.

Open WebUI uses a local administrator account to control access. The account credentials are stored locally on your hardware platform.

When the app opens in your browser, create your admin account as follows:

• Select Get Started at the bottom of the screen.
• Complete the admin account creation with easily remembered details.
• Select Create Admin Account to complete.

Select a model to download

Do the following in the Open WebUI application:

• Click Select a model in the top left corner of the Open WebUI interface.
• Type gpt-oss:20b in the search field.
• Click the Pull "gpt-oss:20b" from Ollama.com button that appears.
• Wait for the model to fully download. You can monitor progress in the interface.

Alternatively, you can enter qwen3.6:latest instead of gpt-oss:20b.

After the download completes, the model appears in the Select a model menu.

Load the model and submit a query

• Select the model from the Select a model menu in the top-left corner.
• In the chat box, enter a prompt such as Write me a haiku about GPUs and press Enter.

Stop Open WebUI with NVIDIA Sync

When you finish your session, you can stop the Open WebUI container from the NVIDIA Sync application window.

• Click on the NVIDIA Sync icon in your system tray or taskbar to open the main application window.
• Under the Custom section, click the x icon on the right of the Open WebUI entry.
• This closes the tunnel and stops the Open WebUI Docker container.

Next steps

You can follow up with other playbooks or use different models.

• For agentic workloads, see the Agent-ready Models tab for the recommended model on your hardware platform.
• Find and compare models from the Ollama model registry.
• Monitor GPU and system usage during inference (for example, via tools available through NVIDIA Sync).

Cleanup

Use these steps when you want to remove Open WebUI from your hardware platform. Cleanup is optional rollback—not required to finish the playbook.

1. Stop the Open WebUI application in the NVIDIA Sync device window (this will also stop the container).

2. Open a terminal on the hardware platform using the Terminal App in the NVIDIA Sync device window.

3. Remove the container with the command:

docker rm open-webui

4. Remove the downloaded image with the command:

docker rmi ghcr.io/open-webui/open-webui:ollama

5. Remove the persistent data volumes with the command:

docker volume rm open-webui open-webui-ollama

6. Remove the custom application from NVIDIA Sync by opening the device window and deleting the Open WebUI entry from the Custom section.