NRTK Tutorial

Introduction

NRTK consists of three main parts:

1. Image Perturbation

2. Perturbation Factories

3. Model Evaluation

The following sections will guide you through setting up and using an example perturber.

By the end of this tutorial, you’ll have a working example that you can expand for your own projects.

Prerequisites

Before starting, ensure the following:

• NRTK is installed (see Installation).

• Software Requirements:

  • Python 3.9+ installed.

  • pip (Python package manager) installed.

• Basic Skills: Familiarity with Python programming and using the terminal or command line.

The following sections will guide you through setting up and using an example perturber.

Image Perturbation

The core of NRTK is based on image perturbation. NRTK offers a wide variety of ways to perturb images. scikit-image, Pillow, openCV, and pyBSM are used for various types of perturbation. The perturbation classes take an image and perform a transformation based on input parameters. The examples shown below focus on a pyBSM based perturber.

For this example, we are going to use the PybsmPerturber from pyBSM. This perturber is useful for creating new images based on existing parameters. The PybsmSensor and PybsmScenario classes contain the parameters for an existing sensor and environment, respectively.

import numpy as np
from pybsm.otf import dark_current_from_density

from nrtk.impls.perturb_image.pybsm.perturber import PybsmPerturber
from nrtk.impls.perturb_image.pybsm.scenario import PybsmScenario
from nrtk.impls.perturb_image.pybsm.sensor import PybsmSensor

opt_trans_wavelengths = np.array([0.58 - 0.08, 0.58 + 0.08]) * 1.0e-6
f = 4  # telescope focal length (m)
p = 0.008e-3  # detector pitch (m)
sensor = PybsmSensor(
    # required
    name="L32511x",
    D=275e-3,  # Telescope diameter (m)
    f=f,
    p_x=p,
    opt_trans_wavelengths=opt_trans_wavelengths,  # Optical system transmission, red  band first (m)
    # optional
    optics_transmission=0.5
    * np.ones(opt_trans_wavelengths.shape[0]),  # guess at the full system optical transmission (excluding obscuration)
    eta=0.4,  # guess
    w_x=p,  # detector width is assumed to be equal to the pitch
    w_y=p,  # detector width is assumed to be equal to the pitch
    int_time=30.0e-3,  # integration time (s) - this is a maximum, the actual integration time will be,
    # determined by the well fill percentage
    dark_current=dark_current_from_density(
        1e-5,
        p,
        p,
    ),  # dark current density of 1 nA/cm2 guess, guess mid range for a silicon camera
    read_noise=25.0,  # rms read noise (rms electrons)
    max_n=96000,  # maximum ADC level (electrons)
    bit_depth=11.9,  # bit depth
    max_well_fill=0.6,  # maximum allowable well fill (see the paper for the logic behind this)
    s_x=0.25 * p / f,  # jitter (radians) - The Olson paper says that its "good" so we'll guess 1/4 ifov rms
    s_y=0.25 * p / f,  # jitter (radians) - The Olson paper says that its "good" so we'll guess 1/4 ifov rms
    da_x=100e-6,  # drift (radians/s) - again, we'll guess that it's really good
    da_y=100e-6,  # drift (radians/s) - again, we'll guess that it's really good
    qe_wavelengths=np.array([0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1]) * 1.0e-6,
    qe=np.array([0.05, 0.6, 0.75, 0.85, 0.85, 0.75, 0.5, 0.2, 0]),
)

scenario = PybsmScenario(
    name="niceday",
    ihaze=1,  # weather model
    altitude=9000.0,  # sensor altitude
    ground_range=0.0,  # range to target
    aircraft_speed=100.0,
)

perturber = PybsmPerturber(sensor=sensor, scenario=scenario, ground_range=10000)

In the example above, we have created a pyBSM perturber where the output image will have a ground_range of 10000m instead of 0m. The image below is the original image we will use for future perturbations.

import os
import urllib.request

import cv2
import matplotlib.pyplot as plt

data_dir = "./pybsm/data"
os.makedirs(data_dir, exist_ok=True)

url = "https://data.kitware.com/api/v1/item/6596fde89c30d6f4e17c9efc/download"

img_path = os.path.join(data_dir, "M-41 Walker Bulldog (USA) width 319cm height 272cm.tiff")
if not os.path.isfile(img_path):
    _ = urllib.request.urlretrieve(url, img_path)  # noqa: S310

img = cv2.imread(img_path)

fig, ax = plt.subplots()
ax.imshow(img)
ax.set_title("Original Image")
ax.set_axis_off()

The code block below shows the loading of the image above and the calling of the perturber. It is important to note that the ground sample distance (or img_gsd) is another parameter the user will have to provide. The resulting image is displayed below the code block.

img_gsd = 3.19 / 165.0  # the width of the tank is 319 cm and it spans ~165 pixels in the image
perturbed_image, _ = perturber.perturb(img, additional_params={"img_gsd": img_gsd})

fig, ax = plt.subplots()
ax.imshow(perturbed_image)
ax.set_title("Perturbed Image")
ax.set_axis_off()

Any of the parameters in either PybsmSensor or PybsmScenario can be modified; however, only one parameter can be modified with one value using the basic perturber. The next section will cover modifying multiple parameters and multiple values.

Perturbation Factories

Continuing on from the previous example, the snippet below shows the initialization of a CustomPybsmPerturbImageFactory. The theta_keys variable controls which parameter(s) we are modifying and thetas are the actual values of the parameter(s). In this example, we are modifying the focal length (f) with the values of 1, 2, and 3. The modified images are displayed below the code block.

from nrtk.impls.perturb_image_factory.pybsm import CustomPybsmPerturbImageFactory

focal_length_pf = CustomPybsmPerturbImageFactory(
    sensor=sensor,
    scenario=scenario,
    theta_keys=["f"],
    thetas=[[1, 2, 3]],
)

_, ax = plt.subplots(1, 3, figsize=(10, 4))
for idx, perturber in enumerate(focal_length_pf):
    perturbed_img, _ = perturber(img, additional_params={"img_gsd": img_gsd})
    ax[idx].set_title(f"focal_length: {idx + 1}")
    ax[idx].imshow(perturbed_img)
    _ = ax[idx].axis("off")
plt.tight_layout()

Not only can you modify multiple values on one parameter, but you can also modify multiple parameters at the same time. The code block below shows the focal length and ground range variables being modified. The resulting images are displayed below the code block.

import itertools

f_groung_range_pf = CustomPybsmPerturbImageFactory(
    sensor=sensor,
    scenario=scenario,
    theta_keys=["f", "ground_range"],
    thetas=[[1, 2], [10000, 20000]],
)
perturber_factory_config = f_groung_range_pf.get_config()
if "theta_keys" in perturber_factory_config:  # pyBSM doesn't follow interface rules
    perturb_factory_keys = perturber_factory_config["theta_keys"]
    thetas = f_groung_range_pf.thetas
else:
    perturb_factory_keys = [f_groung_range_pf.theta_key]
    thetas = [f_groung_range_pf.thetas]
perturber_combinations = [dict(zip(perturb_factory_keys, v)) for v in itertools.product(*thetas)]

_, ax = plt.subplots(1, 4, figsize=(10, 4))
for idx, perturber in enumerate(f_groung_range_pf):
    perturbed_img, _ = perturber(img, additional_params={"img_gsd": img_gsd})
    ax[idx].set_title(
        f"focal_length: {perturber_combinations[idx]['f']}\n"
        f"ground_range: {perturber_combinations[idx]['ground_range']}",
    )
    ax[idx].imshow(perturbed_img)
    _ = ax[idx].axis("off")
plt.tight_layout()

Model Evaluation

To see examples of image classification and object detection, the coco_scorer notebook from the examples directory shows different scoring techniques. For examples of model response to image degradations, there are two notebooks to check out. The simple_generic_generator notebook shows model response to image degradation through perturbers based on scikit-image, Pillow, and openCV. The simple_pybsm_generator notebook shows model response to image degradation through pyBSM-based perturbers.

Next Steps

Now that you’ve completed this tutorial, you can:

• Explore Advanced Features: Try different perturbation methods.

• Use Larger Datasets: Test NRTK on real-world datasets.

See the How-To section for instructions on Perturbation Visualization Examples.

See the Reference section for code documentation.

See the pyBSM documentation for explanatory information regarding the theory behind perturbations, jitter effects, and the significance of certain parameters.