Testing & Evaluation Guides with MAITE

Many robustness testing workflows benefit from using NRTK alongside other tools such as the JATIC program’s Modular AI Trustworthy Engineering (MAITE) toolbox. While NRTK focuses on realistic image perturbations, MAITE provides a standardized interface for evaluating model performance across a set of test conditions. Using these tools together enables modular, reproducible assessments of AI robustness under simulated operational risks.

The following notebooks showcase how NRTK perturbations can be applied to simulate key operational risks within a testing and evaluation (T&E) workflow. Each notebook illustrates potential impact on model performance, utilizing MAITE as an evaluation harness.

Demonstrating Extreme Illumination Perturbations

Simulate brightness changes and evaluate model responses under lighting variability. View notebook.

Demonstrating Visual Focus Perturbations

Apply blur and focus distortions to test performance degradation from defocus. View notebook.

Demonstrating Fog or Haze Perturbations

Evaluate model robustness under haze-like visibility conditions using synthetic perturbations. View notebook.

Demonstrating Lens Flare Perturbations

Simulate a lens flare effect on an image and analyze its average and worst case effects on model precision. View notebook.

Demonstrating Resolution and Noise Transformations

Explore how camera-specific transformations affect model inputs and predictions. View notebook.

Demonstrating Affine Transformations

Explore how affine transformations affect model inputs and predictions. View notebook.

Demonstrating Atmospheric Turbulence Perturbations

Simulate atmospheric distortion effects and assess their impact on image quality and model inference. View notebook.

Demonstrating Rain/Water Droplet Perturbations

Simulate a rain/water droplet effect and analyze its impact on model inputs and predictions. View notebook.

Demonstrating Radial Distortion Perturbations

Simulate a radial distortion effect and analyze its impact on model inputs and predictions. View notebook.

Combining Perturbations with Saliency Maps

Integrate NRTK perturbations with saliency map generation to visualize how image changes affect model interpretation. View notebook.