---
title: "Multi-modal Inference"
publisher: "nvidia"
type: "playbook"
updated: "2025-10-13T17:53:55.893Z"
description: "Setup multi-modal inference with TensorRT"
canonical: "https://build.nvidia.com/spark/multi-modal-inference.md"
---

# Basic idea

Multi-modal inference combines different data types, such as **text, images, and audio**, within a single model pipeline to generate or interpret richer outputs.  
Instead of processing one input type at a time, multi-modal systems have shared representations that  **text-to-image generation**, **image captioning**, or **vision-language reasoning**.  

On GPUs, this enables **parallel processing across modalities** for faster, higher-fidelity results for tasks that combine language and vision.

# What you'll accomplish

You'll deploy GPU-accelerated multi-modal inference capabilities on NVIDIA Spark using TensorRT to run 
Flux.1 and SDXL diffusion models with optimized performance across multiple precision formats (FP16, 
FP8, FP4).

# What to know before starting

- Working with Docker containers and GPU passthrough
- Using TensorRT for model optimization
- Hugging Face model hub authentication and downloads  
- Command-line tools for GPU workloads
- Basic understanding of diffusion models and image generation

# Prerequisites

- NVIDIA Spark device with Blackwell GPU architecture
- Docker installed and accessible to current user
- NVIDIA Container Runtime configured
- Hugging Face account with access to Black Forest Labs models [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) and [FLUX.1-dev-onnx](https://huggingface.co/black-forest-labs/FLUX.1-dev-onnx) on Hugging Face
- Hugging Face [token](https://huggingface.co/settings/tokens) configured with access to both FLUX.1 model repositories
- At least 48GB VRAM available for FP16 Flux.1 Schnell operations
- Verify GPU access: `nvidia-smi`
- Check Docker GPU integration: `docker run --rm --gpus all nvcr.io/nvidia/pytorch:25.11-py3 nvidia-smi`

# Ancillary files

All necessary files can be found in the TensorRT repository [here on GitHub](https://github.com/NVIDIA/TensorRT)
- [**requirements.txt**](https://github.com/NVIDIA/TensorRT/blob/main/demo/Diffusion/requirements.txt) - Python dependencies for TensorRT demo environment
- [**demo_txt2img_flux.py**](https://github.com/NVIDIA/TensorRT/blob/main/demo/Diffusion/demo_txt2img_flux.py) - Flux.1 model inference script  
- [**demo_txt2img_xl.py**](https://github.com/NVIDIA/TensorRT/blob/main/demo/Diffusion/demo_txt2img_xl.py) - SDXL model inference script
- **TensorRT repository** - Contains diffusion demo code and optimization tools

# Time & risk

- **Duration**: 45-90 minutes depending on model downloads and optimization steps

- **Risks**: 
- Large model downloads may timeout
- High VRAM requirements may cause OOM errors
- Quantized models may show quality degradation

- **Rollback**: 
- Remove downloaded models from HuggingFace cache
- Then exit the container environment

* **Last Updated:** 12/22/2025
* Upgrade to latest pytorch container version nvcr.io/nvidia/pytorch:25.11-py3
* Add HuggingFace token setup instructions for model access
* Add docker container permission setup instructioins

## More

- [Instructions](/spark/multi-modal-inference/instructions.md)
- [Troubleshooting](/spark/multi-modal-inference/troubleshooting.md)