--- title: "sam2" publisher: "meta" type: "endpoint" updated: "2025-07-20T16:32:18.249Z" description: "SAM 2 is a segmentation model that enables fast, precise selection of any object in any video or image." canonical: "https://build.nvidia.com/meta/sam2" --- # Model Overview ## Description: **SAM 2**, Segment Anything Model, is a segmentation model that enables fast, precise selection of objects in videos and images. Released by Meta Facebook Artificial Intelligence Research (FAIR), SAM 2 produces segmentation masks of the object of interest in a single image or throughout all video frames, even if the object disappears from view. This model can be used for data annotation, object tracking, or segment anything. In addition, SAM 2 provides a flexible prompt-based interface that enables users to identify objects with a click, bounding box, or mask. Users can select one or more objects in any video frame and use extra prompts to adjust the model’s predictions. SAM2’s efficient video processing with streaming inference enables it for use in real-time, streaming video applications. The model's capabilities have been extended to support segmentation via text-based prompts with GroundingDINO, an open vocabulary object detection model that can detect one or multiple objects in a frame based on the text input. ## Third-Party Community Consideration SAM2 model is not owned or developed by NVIDIA. This model has been developed and built to a third-party's requirements for this application and use case. For **SAM 2 Model Card**, refer to **Model, data and annotation cards** section in [SAM 2 paper](https://arxiv.org/abs/2408.00714). GroundingDINO is a NVIDIA model, pre-trained on a wide range of commercial datasets where the annotations were either human generated or pseudo-labeled. For more information, refer to the [Grounding DINO model card](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/grounding_dino). ### License/Terms of Use **GOVERNING TERMS:** Your use of this API is governed by the [NVIDIA API Trial Service Terms of Use](https://assets.ngc.nvidia.com/products/api-catalog/legal/NVIDIA%20API%20Trial%20Terms%20of%20Service.pdf); and the use of this model is governed by the [NVIDIA Community Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-community-models-license/). ## References: + [SAM 2 paper](https://arxiv.org/abs/2408.00714) + [SAM 2 official code repo](https://github.com/facebookresearch/sam2) ## Model Architecture **Architecture Type:** Transformer
**Network Architecture:** SAM2. For more details, refer to **Model** section in [SAM 2 paper](https://arxiv.org/abs/2408.00714).
## Input: * **Input Types** : Image, Video, Integers (Visual Prompts), text
* **Input Formats** : Image - JPEG, PNG ; Video - MP4
* **Input Parameters:** 2D, 3D
* **Other Properties related to Input** : - The visual prompts include (X,Y) points with labels(include/exclude) in the image/video which are selected by the user by clicking on the image/video in the UI. The visual prompts allow the user to select the regions of interest for segmentation in an image or select the regions of interest to be tracked in a video.
- Users can also provide text description to detect and segement the object of interest. ## Output: * **Output Types**: Image, Video, Integers (Segmentation mask) * **Output Format**: Image - JPEG, PNG ; Video - MP4
* **Output Parameters:** 2D, 3D
* **Other Properties related to Output** : - For Image input, image with segmentation mask(s) overlaid on the object(s) of interest is given as output. - For Video input, video with segmentation mask(s) overlaid on the object(s) of interest. - The segmentation mask is a mask corresponding to the input image/video resolution and the background is represented with value 0 and the object(s) of interest are represented with the respective object id.
## Model Version(s): * [sam2_hiera_large](https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt)
* [GroundingDINO Swin-Tiny](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/grounding_dino/files?version=grounding_dino_swin_tiny_commercial_deployable_v1.0) # Datasets: For details about the dataset refer to [SA-V DataSet](https://ai.meta.com/datasets/segment-anything-video/) and **Model, data and annotation cards** section in [SAM 2 paper](https://arxiv.org/abs/2408.00714).
## Training and Testing Datasets:
**Link**
[SA-V DataSet Download Link ](https://ai.meta.com/datasets/segment-anything-video-downloads/).
**Data Collection Method by dataset**
* Humans: Videos were collected by crowdworkers with unknown equipment via a contracted third-party vendor.
**Labeling Method by dataset**
* Hybrid: Human and Automatic. Masks generated by the Meta Segment Anything Model 2 (SAM 2) and human annotators.
**Properties (Quantity, Dataset Descriptions, Sensor(s))** * SA-V dataset consists of 51K diverse videos and 643K spatio-temporal segmentation masks (i.e., masklets). The videos vary in subject matter. Common themes of the videos include: locations, objects, scenes. Masks range from large scale objects such as buildings to fine grained details such as interior decorations.
## Ethical Considerations: NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/). ## Prototype ```python import os import sys import uuid import zipfile import time import requests import json nvai_url="https://ai.api.nvidia.com/v1/cv/meta/sam2" nvai_polling_url = "https://api.nvcf.nvidia.com/v2/nvcf/pexec/status/" header_auth = f"Bearer {os.getenv('NVIDIA_API_KEY')}" UPLOAD_ASSET_TIMEOUT = 300 # Timeout (in secs) to upload asset MAX_RETRIES = 50 # Max num of retries while polling DELAY_BTW_RETRIES = 2 # adding 1s delay between each polls def _upload_asset(input, description): assets_url = "https://api.nvcf.nvidia.com/v2/nvcf/assets" headers = { "Authorization": header_auth, "Content-Type": "application/json", "accept": "application/json", } s3_headers = { "x-amz-meta-nvcf-asset-description": description, "content-type": "video/mp4", } payload = {"contentType": "video/mp4", "description": description} response = requests.post(assets_url, headers=headers, json=payload, timeout=30) response.raise_for_status() asset_url = response.json()["uploadUrl"] asset_id = response.json()["assetId"] response = requests.put( asset_url, data=input, headers=s3_headers, timeout=UPLOAD_ASSET_TIMEOUT, ) response.raise_for_status() return uuid.UUID(asset_id) if __name__ == "__main__": """Uploads a video or image of your choosing to the NVCF API and sends a request to the SAM2 model. The response is saved to a local directory. Note: You must set up an environment variable, NVIDIA_API_KEY. """ if len(sys.argv) != 4: print("Usage: python test.py ") print("Prompt should be a JSON string of point prompts of as per below example:") sample_prompt_json = ''' { "prompts": [ { "type": "points", "object_id": 1, "frame_index": 0, "points": [ {"x": 20, "y": 375, "label": true}, {"x": 54, "y": 362, "label": false} ] }, { "type": "points", "object_id": 2, "frame_index": 0, "points": [ {"x": 104, "y": 381,"label": true}, {"x": 109, "y": 437, "label": true}, {"x": 77, "y": 377, "label": false}] } ] } ''' print(f"{sample_prompt_json}") sys.exit(1) if len(os.getenv('NVIDIA_API_KEY')) == 0: print("Please set NVIDIA_API_KEY environment variable.") sys.exit(1) asset_id = _upload_asset(open(sys.argv[2], "rb"), "Input Video") point_prompts = json.loads(f"{sys.argv[1]}")["prompts"] inputs = { "model": "meta/sam2-hiera-large", "messages": [ { "role": "user", "content": [ { "type": "media_url", "media_url": { "url": f"data:video/mp4;asset_id,{asset_id}" } } ] } ], "add_objects": False } for point_prompt in point_prompts: inputs["messages"][0]["content"].append(point_prompt) asset_list = f"{asset_id}" headers = { "Content-Type": "application/json", "NVCF-INPUT-ASSET-REFERENCES": asset_list, "NVCF-FUNCTION-ASSET-IDS": asset_list, "Authorization": header_auth, } print(f"Input message: {inputs}") print("Sending request, please wait for response ..") response = requests.post(nvai_url, headers=headers, json=json.loads(json.dumps(inputs))) print(response) if response.status_code != 200 and response.status_code != 202 and response.status_code != 302: sys.exit(1) if response.status_code == 200: # evaluation complete, output video ready with open(f"{sys.argv[3]}.zip", "wb") as out: out.write(response.content) with zipfile.ZipFile(f"{sys.argv[3]}.zip", "r") as z: z.extractall(sys.argv[3]) elif response.status_code == 202: # pending evaluation print("Pending evaluation ...") nvcf_reqid = response.headers['NVCF-REQID'] nvai_polling_url = nvai_polling_url + nvcf_reqid # Polling to check if the response is ready while( MAX_RETRIES ): print(f'Polling ...') headers_polling = { "accept": "application/json", "Authorization": header_auth } response_polling = requests.get(nvai_polling_url, headers=headers_polling) if response_polling.status_code == 202: # evaluation pending print('Result is not yet ready.') MAX_RETRIES -= 1 time.sleep(DELAY_BTW_RETRIES) continue elif response_polling.status_code == 200: # evaluation complete, output video ready print('Result ready!') with open(f"{sys.argv[3]}.zip", "wb") as out: out.write(response_polling.content) break else: print(f"Unexpected response status: {response_polling.status_code}") with zipfile.ZipFile(f"{sys.argv[3]}.zip", "r") as z: z.extractall(sys.argv[3]) print(f"Output saved to {sys.argv[3]}") print(os.listdir(sys.argv[3])) ``` ```javascript const fs = require('fs'); const decompress = require('decompress'); const { setTimeout } = require('timers/promises'); const nvai_url="https://ai.api.nvidia.com/v1/cv/meta/sam2" const nvai_polling_url = "https://api.nvcf.nvidia.com/v2/nvcf/pexec/status/" const header_auth = `Bearer ${process.env.NVIDIA_API_KEY}`; // Constants for polling const MAX_RETRIES = 50; const DELAY_BTW_RETRIES = 2000; // in milliseconds (1 second) async function _upload_asset(input, description) { const assets_url = "https://api.nvcf.nvidia.com/v2/nvcf/assets"; const headers = { "Authorization": header_auth, "Content-Type": "application/json", "accept": "application/json", }; const s3_headers = { "x-amz-meta-nvcf-asset-description": description, "content-type": "video/mp4", }; const payload = { "contentType": "video/mp4", "description": description }; const response = await fetch( assets_url, { method: 'POST', body: JSON.stringify(payload), headers: headers } ); const data = await response.json(); const asset_url = data["uploadUrl"]; const asset_id = data["assetId"]; const fileData = fs.readFileSync(input); await fetch( asset_url, { method: 'PUT', body: fileData, headers: s3_headers } ); return asset_id.toString(); } (async () => { if (process.argv.length != 5) { console.log("Usage: node test.js "); console.log("Prompt should be a JSON string of point prompts of as per below example:") sample_prompt_json = `{ "prompts": [ { "type": "points", "object_id": 1, "frame_index": 0, "points": [ {"x": 20, "y": 375, "label": true}, {"x": 54, "y": 362, "label": false} ] }, { "type": "points", "object_id": 2, "frame_index": 0, "points": [ {"x": 104, "y": 381,"label": true}, {"x": 109, "y": 437, "label": true}, {"x": 77, "y": 377, "label": false}] } ] }` console.log(`${sample_prompt_json}`) process.exit(1); } // Upload specified user asset const asset_id = await _upload_asset(`${process.argv[3]}`, "Input Video"); const point_prompts_message = JSON.parse(`${process.argv[2]}`) const point_prompts = point_prompts_message["prompts"] //const point_prompts = JSON.parse(`${process.argv[2]}`).get("prompts") // Metadata for the request var inputs = { "model": "meta/sam2-hiera-large", "messages": [ { "role": "user", "content": [ { "type": "media_url", "media_url": { "url": `data:video/mp4;asset_id,${asset_id}` } } ] } ], "add_objects": false } for (var i = 0; i < point_prompts.length; i++) { inputs["messages"][0]["content"].push(point_prompts[i]) } const asset_list = asset_id; const headers = { "Content-Type": "application/json", "NVCF-INPUT-ASSET-REFERENCES": asset_list, "NVCF-FUNCTION-ASSET-IDS": asset_list, "Authorization": header_auth }; // Make the request to nvcf console.log("Input message:\n", inputs) console.log(`Sending request, please wait for response ..`) const response = await fetch(nvai_url, { method: 'POST', body: JSON.stringify(inputs), headers: headers }); if (response.status === 200) { // Evaluation complete, output ready // Gather the binary response data const arrayBuffer = await response.arrayBuffer(); const buffer = Buffer.from(arrayBuffer); const zipname = `${process.argv[4]}.zip`; fs.writeFileSync(zipname, buffer); // Unzip the response synchronously await decompress(zipname, process.argv[4]); // Log the output directory and its contents console.log(`Response saved to ${process.argv[4]}`); console.log(fs.readdirSync(process.argv[4])); } else if (response.status === 202) { // Pending evalutation console.log("Pending evaluation ..."); const nvcf_reqid = response.headers.get('NVCF-REQID'); let retries = MAX_RETRIES; while (retries > 0) { console.log('Polling ...'); const pollingResponse = await fetch(`${nvai_polling_url}${nvcf_reqid}`, { method: 'GET', headers: { "accept": "application/json", "Authorization": header_auth } }); if (pollingResponse.status === 202) { // Evaluation still pending console.log('Result is not yet ready.'); retries -= 1; await setTimeout(DELAY_BTW_RETRIES); } else if (pollingResponse.status === 200) { // Evaluation complete, output ready console.log('Result ready!'); const arrayBuffer = await pollingResponse.arrayBuffer(); const buffer = Buffer.from(arrayBuffer); const zipname = `${process.argv[4]}.zip`; fs.writeFileSync(zipname, buffer); // Unzip the response synchronously await decompress(zipname, process.argv[4]); // Log the output directory and its contents console.log(`Response saved to ${process.argv[4]}`); console.log(fs.readdirSync(process.argv[4])); break; } else { console.error('Unexpected response status:', pollingResponse.status); break } if (retries === 0) { console.error('Max retries reached. Evaluation not completed.'); } } } else { console.error('Unexpected response status:', response.status); } })(); ``` ```bash #!/bin/bash set -e # Check arguments if [ "$#" -ne 3 ]; then echo "Usage: ./test.sh " echo "Prompt should be a JSON string of point prompts of as per below example:" echo '{ "type": "points", "object_id": 1, "frame_index": 0, "points": [ {"x": 20, "y": 375, "label": true}, {"x": 54, "y": 362, "label": false} ] }, { "type": "points", "object_id": 2, "frame_index": 0, "points": [ {"x": 104, "y": 381,"label": true}, {"x": 109, "y": 437, "label": true}, {"x": 77, "y": 377, "label": false}] }' exit 1 fi # Constants MAX_RETRIES=50 DELAY_BTW_RETRIES=2 # in seconds header_auth="Bearer $NVIDIA_API_KEY" assets_url="https://api.nvcf.nvidia.com/v2/nvcf/assets" nvai_url="https://ai.api.nvidia.com/v1/cv/meta/sam2" nvai_polling_url="https://api.nvcf.nvidia.com/v2/nvcf/pexec/status/" function upload_asset { local input=$1 local description=$2 response=$(curl -s -X POST $assets_url \ -H "Authorization: ${header_auth}" \ -H "Content-Type: application/json" \ -H "accept: application/json" \ -d '{"contentType": "video/mp4", "description": "'"${description}"'"}') asset_url=$(echo ${response} | jq -r .uploadUrl) asset_id=$(echo ${response} | jq -r .assetId) curl -s -X PUT -H "x-amz-meta-nvcf-asset-description: ${description}" -H "content-type: video/mp4" \ --data-binary "@${input}" ${asset_url} echo ${asset_id} } if [ -z "${NVIDIA_API_KEY}" ]; then echo "Please set NVIDIA_API_KEY." fi asset_id=$(upload_asset "$2" "Input Video") echo "asset_id: ${asset_id}" inputs='{"model": "meta/sam2-hiera-large", "messages": [{"role": "user", "content": ['"${1}"', {"type": "media_url", "media_url": {"url": "data:video/mp4;asset_id,'"${asset_id}"'" }}]}], "add_objects": false}' asset_list="${asset_id}" # The response will include a location header that curl doesn't trivially # redirect to due to its complexity. We need to extract the location header # and then download the file from that location. echo "Sending request, please wait for response .." response=$(curl -D - -s -w "%{http_code}" -X POST $nvai_url \ -H "Content-Type: application/json" \ -H "NVCF-INPUT-ASSET-REFERENCES: ${asset_list}" \ -H "NVCF-FUNCTION-ASSET-IDS: ${asset_list}" \ -H "Authorization: ${header_auth}" \ -d "${inputs}") echo "${response}" # Extracting status code status_code=$(echo "$response" | grep HTTP | awk '{print $2}') if [[ "${status_code}" != "200" && "${status_code}" != "202" && "${status_code}" != "302" ]]; then exit 1 fi # Remove any newlines, carriage returns, spaces, quotes, and commas from the location header location=$(echo "$response" | grep -i location | awk '{print $2}' | tr -d '\n' | tr -d '\r' | tr -d ' ' | tr -d '"' | tr -d ',') if [ -n "$location" ]; then # location non empty, Evaluation complete, download the file curl -s "$location" -o "$3.zip" # Unzip the file unzip -q "$3.zip" -d "$3" echo "Response saved to $3.zip" echo $(ls "$3") exit 0 elif [ -z "$location" ]; then # location empty, Pending evaluation, need polling echo "Pending evaluation ..." retries=$MAX_RETRIES while [ $retries -gt 0 ]; do echo "Polling ..." nvcf_reqid=$(echo "$response" | grep -i nvcf-reqid | awk '{print $2}' | tr -d '\n' | tr -d '\r' | tr -d ' ' | tr -d '"' | tr -d ',') polling_url="${nvai_polling_url}${nvcf_reqid}" polling_response=$(curl -D - -s -w "%{http_code}" -X GET $polling_url \ -H "accept: application/json" \ -H "Authorization: ${header_auth}") # Extracting location polling_location=$(echo "$polling_response" | grep -i location | awk '{print $2}' | tr -d '\n' | tr -d '\r' | tr -d ' ' | tr -d '"' | tr -d ',') if [ -z "$polling_location" ]; then echo "Result is not yet ready." retries=$((retries - 1)) sleep $DELAY_BTW_RETRIES elif [ -n "$polling_location" ]; then echo "Result ready!" curl -s "$polling_location" -o "$3.zip" unzip -q "$3.zip" -d "$3" echo "Response saved to $3.zip" echo $(ls "$3") exit 0 else echo "Unexpected response status: $polling_status" exit 1 fi done if [ $retries -eq 0 ]; then echo "Max retries reached. Evaluation not completed." exit 1 fi else echo "Unexpected response status: $status_code" exit 1 fi ```