Testing & Evaluation Guides with MAITE#
Many robustness testing workflows benefit from using NRTK alongside other tools such as the JATIC program’s Modular AI Trustworthy Engineering (MAITE) toolbox. While NRTK focuses on realistic image perturbations, MAITE provides a standardized interface for evaluating model performance across a set of test conditions. Using these tools together enables modular, reproducible assessments of AI robustness under simulated operational risks.
The following notebooks showcase how NRTK perturbations can be applied to simulate key operational risks within a testing and evaluation (T&E) workflow. Each notebook illustrates potential impact on model performance, utilizing MAITE as an evaluation harness.
Photometric Risks#
Perturbations that alter pixel intensity, color balance, or lighting conditions.
Simulate brightness changes and evaluate model responses under lighting variability.
Simulate a lens flare effect on an image and analyze its average and worst case effects on model precision.
Geometric Risks#
Perturbations that change spatial layout through rotation, scaling, or translation.
Explore how affine transformations affect model inputs and predictions.
Environment Risks#
Perturbations that replicate weather and atmospheric visibility conditions.
Evaluate model robustness under haze-like visibility conditions using synthetic perturbations.
Simulate a rain/water droplet effect and analyze its impact on model inputs and predictions.
Optical Risks#
Perturbations from sensor optics, camera physics, and atmospheric distortion.
Apply blur and focus distortions to test performance degradation from defocus.
Explore how camera-specific transformations affect model inputs and predictions.
Simulate camera motion jitter and assess its impact on image quality and model inference.
Simulate atmospheric distortion effects and assess its impact on image quality and model inference.
Simulate a radial distortion effect and analyze its impact on model inputs and predictions.