# Visual ChangeNet Segmentation Model Card (Commercial) ## Model Overview The Visual ChangeNet-Segmentation Model detects changes in land cover using remote sensing imagery (RSI). This model is ready for commercial use. ## References: ### Using TAO Pre-trained Models - Get [TAO Container](https://ngc.nvidia.com/catalog/containers/nvidia:tao:tao-toolkit) - Get other purpose-built models from the NGC model registry: - [TrafficCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:trafficcamnet) - [PeopleNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplenet) - [PeopleNet-Transformer](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplenet_transformer) - [DashCamNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:dashcamnet) - [FaceDetectIR](https://ngc.nvidia.com/catalog/models/nvidia:tao:facedetectir) - [VehicleMakeNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:vehiclemakenet) - [VehicleTypeNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:vehicletypenet) - [PeopleSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplesegnet) - [PeopleSemSegNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:peoplesemsegnet) - [License Plate Detection](https://ngc.nvidia.com/catalog/models/nvidia:tao:lpdnet) - [License Plate Recognition](https://ngc.nvidia.com/catalog/models/nvidia:tao:lprnet) - [PoseClassificationNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:poseclassificationnet) - [Facial Landmark](https://ngc.nvidia.com/catalog/models/nvidia:tao:fpenet) - [FaceDetect](https://ngc.nvidia.com/catalog/models/nvidia:tao:facenet) - [2D Body Pose Estimation](https://ngc.nvidia.com/catalog/models/nvidia:tao:bodyposenet) - [ActionRecognitionNet](https://ngc.nvidia.com/catalog/models/nvidia:tao:actionrecognitionnet) - [People ReIdentification](https://ngc.nvidia.com/catalog/models/nvidia:tao:reidentificationnet) - [PointPillarNet](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pointpillarnet) - [CitySegFormer](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/citysemsegformer) - [Retail Object Detection](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/retail_object_detection) - [Retail Object Embedding](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/retail_object_recognition) - [Optical Inspection](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/optical_inspection) - [Optical Character Detection](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/ocdnet) - [Optical Character Recognition](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/ocrnet) - [PCB Classification](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pcb_classification) - [PeopleSemSegFormer](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/peoplesemsegformer) ## Model Architecture: **Architecture Type:** Transformer-Based
**Network Architecture:** Siamese Network
Visual ChangeNet is a state of the art transformer-based Change Detection model. Visual ChangeNet is based on Siamese Network, which is a class of neural network architectures containing two or more identical subnetworks. The training algorithm works by updating the parameters across all the sub-networks in tandem. In TAO, Visual ChangeNet supports two images as input where the end goal is to either classify or segment the change between the "golden or reference" image and the "test" image. More specifically, this model was trained with the [NVDINOv2 backbone](https://arxiv.org/abs/2304.07193), which was trained using a self-supervised manner on NVIDIA proprietary data and achieved SOTA accuracy on zero-shot ImageNet classification. To enable the ViT backbone into Visual ChangeNet, the [ViT-Adapter](https://arxiv.org/abs/2205.08534) was used as the neck architecture. The ViT-Adapter improves the accuracy on dense predictions, such as object detection and segmentation. In TAO, two different types of Change Detection networks are supported: * Visual ChangeNet-Segmentation - for segmentation of change between two input images. * Visual ChangeNet-Classification - for classification of change between two input images. Visual ChangeNet-Segmentation is specifically intended for change segmentation. In this model card, the Visual ChangeNet-Segmentation model is leveraged to demonstrate land cover semantic change detection using the LandSat-SCD dataset. The model uses a pretrained NVDINOv2 backbone, trained on NVIDIA-commercial dataset, and then fine-tuned on the LandSat-SCD dataset. ## Input: **Input Type(s):** Images
**Input Format(s):** Red, Green, Blue (RGB)
**Input Parameters:** Three-Dimensional (3D)
**Other Properties Related to Input:**
Two input images: * Golden: RGB Image of dimensions: 416 X 416 X 3 (H x W x C) * Sample: RGB Image of dimensions: 416 X 416 X 3 (H x W x C) Channel Ordering of the Input: NCHW, where N = Batch Size, C = number of channels (3), H = Height of images (416), W = Width of the images (416) Here is a sample image for a pre and post change images along with ground-truth segmentation change map side-by-side. Land cover change

## Output: **Output Type(s):** Segmentation Change Map
**Output Format:** 3D Vector
**Other Properties Related to Output:**
Segmentation change map with the same resolution as the input images: 416 X 416 X 10 (H x W x C), where C = number of output change classes. ## Software Integration: **Runtime Engine(s):** * TAO - 5.2
**Supported Hardware Architecture(s):**
* Ampere
* Jetson
* Hopper
* Lovelace
* Pascal
* Turing
* Volta
**Supported Operating System(s):**
* Linux
* Linux 4 Tegra
## Model Version(s): - **trainable_v1.0** - NVDINOv2 Visual ChangeNet-Segmentation model LandSat-SCD is trainable. - **deployable_v1.0** - NVDINOv2 Visual ChangeNet-Segmentation model LandSat-SCD is deployable to DeepStream. - **deployable_v1.1** - NVDINOv2 Visual ChangeNet-Segmentation model LandSat-SCD is deployable to DeepStream. - **deployable_v1.2** - NVDINOv2 Visual ChangeNet-Segmentation model LandSat-SCD is deployable to DeepStream. ## Training & Evaluation: ## Training Dataset: **Data Collection Method by dataset:**
* Automatic/Sensors
**Labeling Method by dataset:**
* Human
**Properties:**
Trained on the open-source remote sensing semantic land change detection dataset consisting of 8468 images containing remote sensing (RS) image pairs of resolution 416 × 416. They are randomly split into three parts to make train, val, and test sets of samples 6053, 1729, and 686 respectively. | Dataset | No. of images | |--|--| |LandSat-SCD| 8468 | ## Evaluation Dataset: **Data Collection Method by dataset:**
* Automatic/Sensors
**Labeling Method by dataset:**
* Human
**Properties:**
Evaluated on the open-source remote sensing semantic land change detection dataset of 686 images. #### Methodology and KPI The performance of the Visual ChangeNet-Segmentation model for multi-class semantic change detection is measured using overall accuracy, average precision, average recall and avergae IoU score for all the classes. | Model | Model Architecture | Testing Images | Precision | Recall | IoU | F1 | Overall Accuracy | |--|--|--|--|--|--|--|--| | Visual ChangeNet-Segmentation | Siamese Network | 686 | 93 | 92.2 | 86.5 | 92.58 | 97.85 | ## Inference: **Engine:** Tensor(RT)
**Test Hardware:**
- Jetson AGX Xavier - Xavier NX - Orin - Orin NX - NVIDIA T4 - Ampere GPU - A2 - A30 - L4 - T4 - DGX H100 - DGX A100 - DGX H100 - L40 - JAO 64GB - Orin NX16GB - Orin Nano 8GB The inference is run on the provided unpruned model at FP16 precision. The inference performance is run using [`trtexec`](https://github.com/NVIDIA/TensorRT/tree/main/samples/trtexec) on Jetson AGX Xavier, Xavier NX, Orin, Orin NX and NVIDIA T4, and Ampere GPUs. The Jetson devices are running at Max-N configuration for maximum GPU frequency. The performance shown here is the inference only performance. The end-to-end performance with streaming video data might vary depending on other bottlenecks in the hardware and software. NVDINOv2 + ViT-Adapter + Visual ChangeNet | Platform | BS | FPS | |------------------|-----|--------| | Orin NX 16GB | 16 | 1.5 | | AGX Orin 64GB | 16 | 9.41 | | A2 | 8 | 5.9 | | T4 | 16 | 2.29 | | L4 | 16 | 4.68 | | A30 | 16 | 35.8 | | L40 | 16 | 11.3 | | A100 | 32 | 10.8 | | H100 | 32 | 23.5 | ## Using this Model These models need to be used with NVIDIA hardware and software. For hardware, the models can run on any NVIDIA GPU including NVIDIA Jetson devices. These models can only be used with the [Train Adapt Optimize (TAO) Toolkit](https://developer.nvidia.com/tao-toolkit), or [TensorRT](https://developer.nvidia.com/tensorrt). The primary use case for these models is for Visual ChangeNet-Segmentation using RGB images. The model is a Siamese Network that outputs semantic change maps denoting pixel-level change between the two images. These models are intended for training and fine-tuning using the TAO Toolkit and your datasets for image comparison. High-fidelity models can be trained on new use cases. A Jupyter Notebook is available as a part of the [TAO container](https://ngc.nvidia.com/catalog/containers/nvidia:tao:tao-toolkit) and can be used to re-training. The models are also intended for edge deployment using TensorRT. ### Using the Model with TAO To use these models as pretrained weights for transfer learning, use the following as a template for the `model` and `train` component of the experiment spec file to train a Siamese Network model. For more information on the experiment spec file, see the [TAO Toolkit User Guide - Visual ChangeNet-Segmentation](https://docs.nvidia.com/tao/tao-toolkit/text/visual_changenet/visual_changenet_segment.html). ```yaml model: backbone: type: "vit_large_nvdinov2" pretrained_backbone_path: null freeze_backbone: False ``` ## Technical Blogs - Learn how to transform [Industrial Defect Detection with NVIDIA TAO and Vision AI Models](https://developer.nvidia.com/blog/transforming-industrial-defect-detection-with-nvidia-tao-and-vision-ai-models/) - Read the 2 part blog on training and optimizing 2D body pose estimation model with TAO - [Part 1](https://developer.nvidia.com/blog/training-optimizing-2d-pose-estimation-model-with-tao-toolkit-part-1) | [Part 2](https://developer.nvidia.com/blog/training-optimizing-2d-pose-estimation-model-with-tao-toolkit-part-2) - Learn how to train [real-time license plate detection and recognition app](https://developer.nvidia.com/blog/creating-a-real-time-license-plate-detection-and-recognition-app) with TAO and DeepStream. - Model accuracy is extremely important, learn how you can achieve [state of the art accuracy for classification and object detection models](https://developer.nvidia.com/blog/preparing-state-of-the-art-models-for-classification-and-object-detection-with-tao-toolkit/) using TAO - Learn how to train the [Instance segmentation model using MaskRCNN with TAO](https://developer.nvidia.com/blog/training-instance-segmentation-models-using-maskrcnn-on-tao-toolkit/) - Read the technical tutorial on how [PeopleNet model can be trained with custom data using Transfer Learning Toolkit](https://devblogs.nvidia.com/training-custom-pretrained-models-using-tlt/) - Learn how to [train and deploy real-time intelligent video analytics apps and services using DeepStream SDK](https://devblogs.nvidia.com/building-iva-apps-using-deepstream-5.0/) ## Suggested Reading - More information on about TAO Toolkit and pre-trained models can be found at the [NVIDIA Developer Zone](https://developer.nvidia.com/tao-toolkit) - Read the [TAO Quick Start](https://docs.nvidia.com/tao/tao-toolkit/text/tao_toolkit_quick_start_guide.html) guide and [release notes](https://docs.nvidia.com/tao/tao-toolkit/text/release_notes.html). - If you have any questions or feedback, see the discussions on [TAO Toolkit Developer Forums](https://forums.developer.nvidia.com/c/accelerated-computing/intelligent-video-analytics/tao-toolkit/17) - Deploy your model on the edge using DeepStream. Learn more about [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk) ## Ethical Considerations: NVIDIA Visual ChangeNet-Segmentation model detects changes between pair-wise images. NVIDIA’s platforms and application frameworks enable developers to build a wide array of AI applications. Consider potential algorithmic bias when choosing or creating the models being deployed. NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. For more detailed information on ethical considerations for this model, please see the Model Card++ Promise and the Explainability, Bias, Safety & Security, and Privacy Subcards. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability)

nvidia

visual-changenet