---
title: "kosmos-2"
publisher: "microsoft"
type: "endpoint"
updated: "2024-08-26T16:47:14.247Z"
description: "Groundbreaking multimodal model designed to understand and reason about visual elements in images."
canonical: "https://build.nvidia.com/microsoft/microsoft-kosmos-2"
---

# Model Overview

## Description:

Kosmos-2 model is a groundbreaking multimodal large language model (MLLM). 
Kosmos-2 is designed to ground text to the visual world, 
enabling it to understand and reason about visual elements in images.

## Terms of use

By using this model, you are agreeing to the terms and conditions of the 
[license](https://github.com/microsoft/unilm/blob/master/LICENSE), 
acceptable use policy and Microsoft Research privacy policy.

## References(s):

* [KOSMOS-2 paper](https://arxiv.org/pdf/2306.14824.pdf)

## Model Architecture:
**Architecture Type:**  Transformer <br>
**Network Architecture:** GPT + CLIP <br>

## Input:
**Input Format:** Red, Green, Blue (RGB) Image + Text <br>
**Input Parameters:** Temperature, TopP <br>
**Other Properties Related to Input:** None <br>

## Output: <br>
**Output Format:** Text <br>
**Output Parameters:** Max output tokens, Bounding boxes <br>
**Other Properties Related to Output:** None <br>

## Supported Operating System(s):
Linux

# Inference:
**Engine:** [Triton](https://developer.nvidia.com/triton-inference-server) <br>
**Test Hardware:** Other <br>

## Prototype

```python
import requests

invoke_url = "https://ai.api.nvidia.com/v1/vlm/microsoft/kosmos-2"

headers = {
"Authorization": "Bearer ",
"Accept": "application/json",
}

payload = {
"messages": [
{
"role": "user",
"content": ""
}
]
}

# re-use connections
session = requests.Session()

response = session.post(invoke_url, headers=headers, json=payload)

response.raise_for_status()
response_body = response.json()
print(response_body)
```

```javascript
import fetch from "node-fetch";

const invokeUrl = "https://ai.api.nvidia.com/v1/vlm/microsoft/kosmos-2"

const headers = {
"Authorization": "Bearer ",
"Accept": "application/json",
}

const payload = {
"messages": [
{
"role": "user",
"content": ""
}
]
}

let response = await fetch(invokeUrl, {
method: "post",
body: JSON.stringify(payload),
headers: { "Content-Type": "application/json", ...headers }
});

let response_body = await response.json()

console.log(JSON.stringify(response_body))
```

```bash
invoke_url='https://ai.api.nvidia.com/v1/vlm/microsoft/kosmos-2'

authorization_header='Authorization: Bearer '
accept_header='Accept: application/json'
content_type_header='Content-Type: application/json'

data=$'{
"messages": [
{
"role": "user",
"content": ""
}
]
}'

response=$(curl --silent -i -w "\n%{http_code}" --request POST \
--url "$invoke_url" \
--header "$authorization_header" \
--header "$accept_header" \
--header "$content_type_header" \
--data "$data"
)

echo "$response"
```