nvidia

LLM Router

Route LLM requests to the best model for the task at hand.

Today's large language models (LLMs) are subject to a trade-off between reasoning capabilities and computational efficiency. While powerful models excel at complex reasoning tasks, sophisticated test-time compute, and level 2 thinking (reasoning about their own reasoning), they’re computationally expensive and slower, making them impractical for simpler tasks. The NVIDIA AI Blueprint for an LLM router is designed to mitigate this trade-off by intelligently directing prompts to the most appropriate model, ensuring optimal balance between reasoning depth and computational efficiency. Through its lightweight classification models that run in milliseconds, it routes simple queries to fast, efficient models and directs prompts that demand careful analysis and self-reflective reasoning to more powerful models that can apply extensive test-time computation.

The blueprint achieves this through a flexible architecture that supports multiple routing strategies, from task-based classification to user-intent analysis to reasoning-based routing. Using specialized classification models, it analyzes each prompt for complexity, required domain knowledge, and need for iterative thinking, enabling organizations to maintain high-quality responses for complex reasoning tasks while optimizing computational resources. This strategic routing lets organizations scale their AI systems efficiently and ensure deep reasoning capabilities are available when needed, fundamentally transforming how we deploy and utilize language models in production environments.

Architecture Diagram

What’s Included in the Blueprint

Key Features

This reference architecture includes an architectural diagram, an NVIDIA Brev launchable with a Jupyter notebook for rapid exploration and experimentation, and source code for local deployment and customization. The LLM Router supports the following key features and components:

Low-Latency Router Controller: To manage routing logic and decision-making for optimal query distribution. The router models are very small, so they don’t add extra latency and are easily fine-tunable.
Response Evaluation Strategies: For assessing LLM outputs to improve routing accuracy and decisions.
Multi-LLM Routing: For routing among multiple models, going beyond binary selection.
Customization Workflows: For fine-tuning router models for specific use cases.
Flexible Routing Methodologies: For cost-based, response quality-based, task-based, and intent-based routing.
Modular Design: Can deploy either the router controller with the router server or only the router server alone and use a different proxy. The router controller and router server can be deployed in separate systems. The router server needs a GPU and the router controller doesn't need a GPU.
Powerful Abstraction Layer: Streamlines deployment by handling routing pipelines and model orchestration behind the scenes.

Software Used in This Blueprint

NVIDIA NIM™ microservices

Other

NVIDIA Triton Inference Server

Minimum System Requirements

Hardware Requirements

Any NVIDIA GPU with an NVIDIA architecture newer than Volta™ (V100), such as Turing™ (T4), Ampere™ (A100, RTX 30 series), Hopper™ (H100), or later.

Software Requirements

Git LFS
Docker
Docker Compose
NVIDIA API key from build.nvidia.com (see instructions)

Ethical Considerations

NVIDIA believes trustworthy AI is a shared responsibility, and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their supporting model team to ensure the models meet requirements for the relevant industry and use case and address unforeseen product misuse. For more detailed information on ethical considerations for the models, please see the Model Card++ Explainability, Bias, Safety and Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI concerns here.

License

Use of the models in this blueprint is governed by the NVIDIA AI Foundation Models Community License.

Warning: The Terms of Use section below is a work in progress and will be updated with the final terms.

Terms of Use

GOVERNING TERMS: The software is governed by the NVIDIA Software License Agreement and Product-Specific Terms for NVIDIA AI Products. Use of the Complexity and Task Qualifier model is governed by the NVIDIA Open Model License Agreement. Additional Information: MIT License.

Use of these model is governed by the NVIDIA AI Foundation Models Community License Agreement. ADDITIONAL INFORMATION: Llama 3.1 Community License Agreement, Built with Llama;