NVIDIA Maxine Audio2Face-2D NIM uses gRPC APIs for inferencing requests. A NGC API KEY is required to download the appropriate models and resources when starting the NIM. Pass the value of the API key to the `docker run` command in the next section as the `NGC_API_KEY` environment variable as indicated. If you are not familiar with how to create the `NGC_API_KEY` environment variable, the simplest way is to export it in your terminal: ```bash export NGC_API_KEY= ``` Run one of the following commands to make the key available at startup: ```bash # If using bash echo "export NGC_API_KEY=" >> ~/.bashrc # If using zsh echo "export NGC_API_KEY=" >> ~/.zshrc ``` Other, more secure options include saving the value in a file, so that you can retrieve with `cat $NGC_API_KEY_FILE`, or using a [password manager](https://www.passwordstore.org/). To pull the NIM container image from NGC, first authenticate using the NVIDIA Container Registry with the following command: ```bash echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin ``` Use `$oauthtoken` as the username and `NGC_API_KEY` as the password. The `$oauthtoken` username is a special keyword that indicates that you will authenticate with an API key and not a user name and password. The following command launches the Maxine Audio2Face-2D NIM container with the gRPC service. Find reference to runtime parameters for the container [here](https://docs.nvidia.com/nim/maxine/audio2face-2d/latest/getting-started.html#runtime-parameters-for-the-container). ```bash # Choose manifest profile id based on target architecture. export MANIFEST_PROFILE_ID= ``` Then run the NIM launch command ```bash docker run -it --rm --name=maxine-audio2face-2d-nim \ --net host \ --runtime=nvidia \ --gpus all \ --shm-size=8GB \ -e NGC_API_KEY=$NGC_API_KEY \ -e NIM_MANIFEST_PROFILE=$MANIFEST_PROFILE_ID \ -e NIM_HTTP_API_PORT=9000 \ -e NIM_GRPC_API_PORT=50051 \ -p 9000:9000 \ -p 50051:50051 \ nvcr.io/nim/nvidia/maxine-audio2face-2d:latest ``` ```{note} The flag `--gpus all` is used to assign all available GPUs to the NIM container. To assign specific GPU to the NIM container (in case of multiple GPUs available in your machine) use `--gpus '"device=0,1,2..."'` ``` ### Model Manifest Profiles | GPU Architecture (compute capability) | Manifest profile ID | | ---------------- | ---------------- | | Ada (cc 8.9) | 1c3bc7246b69c0bb678612b19cf464e002e943828d0273212cbf5e5775fbae69 | | Ampere (cc 8.6) | 4e65acd61da72c9a24a1156b9fc9736a257db49a675944ac2cf1913f555301a8 | | Ampere (cc 8.0) | 92d71e45910e3ec5442d480bbdd72c5c0d4ee4ffe63631b3391b54c35806c159 | | Turing (cc 7.5) | 65e48f31f1e626fac0b04687b1b0820a2d4cc6236c69e27cffb1ca7c653bc187 | | Volta (cc 7.0) | 61ba53d7d857305f1911bf537abefd274b571cbe2b6ab119b3088aa842cf806b | ```{note} Use the correct manifest profile ID from the given table above in the NIM launch command. If incorrect manifest profile ID is used, then on inference a deserialization error will occur. If no manifest profile ID is provided, then default ID will be used. Default: Ada (compute capability 8.9). ``` If the NIM launch is successful, you will get a response similar to the following. ```bash I1121 09:59:41.919738 49 server.cc:674] +----------------------+---------+--------+ | Model | Version | Status | +----------------------+---------+--------+ | GazeRedirectionKey68 | 1 | READY | | SpeechLPPerfMode1 | 1 | READY | | SpeechLPPerfMode2 | 1 | READY | | SpeechLPPerfMode3 | 1 | READY | | SpeechLPQualMode1 | 1 | READY | | SpeechLPQualMode2 | 1 | READY | | SpeechLPQualMode3 | 1 | READY | +----------------------+---------+--------+ I1121 09:59:42.017374 49 metrics.cc:877] "Collecting metrics for GPU 0: " I1121 09:59:42.020257 49 metrics.cc:770] "Collecting CPU metrics" I1121 09:59:42.020950 49 tritonserver.cc:2598] +----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Option | Value | +----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | server_id | triton | | server_version | 2.50.0 | | server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data parameters statistics trace logging | | model_repository_path[0] | /opt/maxine/models | | model_control_mode | MODE_NONE | | strict_model_config | 0 | | model_config_name | | | rate_limit | OFF | | pinned_memory_pool_byte_size | 268435456 | | cuda_memory_pool_byte_size{0} | 67108864 | | min_supported_compute_capability | 6.0 | | strict_readiness | 1 | | exit_timeout | 30 | | cache_enabled | 0 | +----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I1121 09:59:42.023798 49 grpc_server.cc:2558] "Started GRPCInferenceService at 0.0.0.0:8001" I1121 09:59:42.024109 49 http_server.cc:4704] "Started HTTPService at 0.0.0.0:8000" I1121 09:59:42.065331 49 http_server.cc:362] "Started Metrics Service at 0.0.0.0:8002" Maxine GRPC Service: Listening to 0.0.0.0:8004 ``` ```{note} By default Maxine Audio2Face-2D NIM's gRPC service is hosted on port `8004`. You will have to use this port for inferencing requests. ``` We have provided a sample client script file in our GitHub repo. The script could be used to invoke the Docker container using the following instructions. Download the Maxine Audio2Face-2D client code by cloning the [NVIDIA Maxine NIM Clients Repository](https://github.com/NVIDIA-Maxine/nim-clients): ```bash git clone https://github.com/NVIDIA-Maxine/nim-clients.git cd nim-clients/audio2face-2d/ ``` Install the dependencies for the NVIDIA Maxine Audio2Face-2D client: - For python client ```bash sudo apt-get install python3-pip pip install -r python/requirements.txt ``` - For NodeJS client ```bash # Add the repo and install the latest stable Node.js curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash - sudo apt install nodejs npm # Install all the required packages using package.json file in nodejs directory npm install --prefix nodejs/ ``` ## Running Inference via Python Script You can use the sample client script in the Maxine Audio2Face-2D GitHub repo to send a gRPC request to the hosted NIM server: 1. Go to the python scripts directory ```bash cd python/scripts ``` 2. Run the command to send gRPC request ```bash python audio2face-2d.py --target --audio-input --portrait-input --output --head-rotation-animation-filepath --head-translation-animation-filepath ``` To view details of command line arguments, run this command: ```bash python audio2face-2d.py -h ``` You will get a response similar to the following. ```bash options: -h, --help show this help message and exit --target TARGET IP:port of gRPC service, when hosted locally. --audio-input AUDIO_INPUT The path to the input audio file. --portrait-input PORTRAIT_INPUT The path to the input portrait file. --output OUTPUT The path for the output video file. --head-rotation-animation-filepath HEAD_ROTATION_ANIMATION_FILEPATH The path for the head_rotation_animation.csv file. Only required for HEAD_POSE_MODE_USER_DEFINED_ANIMATION --head-translation-animation-filepath HEAD_TRANSLATION_ANIMATION_FILEPATH The path for the head_translation_animation.csv file. Only required for HEAD_POSE_MODE_USER_DEFINED_ANIMATION ``` If the command line arguments are not passed, the script will use the following default values: - `target` is `127.0.0.1:8004` - `portrait-input` is `../../assets/sample_portrait_image.png` - `audio-input` is `../../assets/sample_audio.wav` - `output` will be the current directory where the output file will be generated with name `output.mp4` - `head-rotation-animation-filepath` is `../../assets/head_rotation_animation.csv` - `head-translation-animation-filepath` is `../../assets/head_translation_animation.csv` ## Running Inference via NodeJS Script NodeJS client, similar to Python client, can also be used to exercise A2F2D feature by sending gRPC requests to hosted NIM server. The `audio2face-2d.js` NodeJS script takes a portrait image and a `wav` or `pcm` audio file (default is wav) and generates the mp4 video output. This script can also be used to stream the output video on the browser while it is received from the server. Go the NodeJS script folder ```bash cd nodejs/scripts ``` Run the following command to send a gRPC request (all command line parameters are optional): ```bash node audio2face-2d.js --target --audio-input --portrait-input --output --format --head-rotation-animation-filepath --head-translation-animation-filepath --browser ``` In case of pcm audio input, the sample rate, channels and the bitDepth are used as 48000, 1 and 16 respectively in the code but these settings need to be updated as per the pcm file config. ### Browser streaming `index.html` file defines a simple page to show the video on browser which is present in the same directory as the script. On the browser, enter the server URL `http://URL_OF_THE_SERVER:3000` , which is `http://127.0.0.1:3000` if testing on the same machine. This should open a web page that will play the output video as it is streamed from the server. To view details of command line arguments run this command ```bash node audio2face-2d.js -h ``` You will get a response similar to the following. ```bash Options: -h, --help Show this help message and exit --target IP:port of gRPC service, when hosted locally. --audio-input The path to the input audio file. (default: "../../assets/sample_audio.wav") --portrait-input The path to the input portrait file. (default: "../../assets/sample_portrait_image.png") --output The path for the output video file. (default: "output.mp4") --head-rotation-animation-filepath The path to the head rotation animation csv file. Only required for HEAD_POSE_MODE_USER_DEFINED_ANIMATION (default: "../../assets/head_rotation_animation.csv") --head-translation-animation-filepath The path to the head translation animation csv file. Only required for HEAD_POSE_MODE_USER_DEFINED_ANIMATION (default: "../../assets/head_translation_animation.csv") --format Audio format - wav or pcm (default: "wav") ``` For more details on getting started with this NIM including configuring using parameters, visit the [NVIDIA Maxine Audio2Face-2D NIM Docs](https://docs.nvidia.com/nim/maxine/audio2face-2d/latest/index.html).

nvidia/audio2face-2d

Input

Output