Interfaces

The NRTK API consists of a number of object-oriented functor interfaces for item-response curve (IRC) generation, namely for assessing model response to perturbations on given input data. These interfaces focus on black-box IRC generation. In addition to the driver, or generator, of this task, there are two other main components: reference image perturbation in preparation for black-box testing and black-box scoring for model outputs. The PeturbImageFactory interface provides the utility to easily vary specified parameters of a particular perturber. The generator will execute upon a given perturber factory as well as a given model and scorer to generate the IRC for given input data. We define a few similar interfaces for performing the IRC generation, separated by the intermediate algorithmic use cases, one for object detection and one for image classification.

We explicitly do not require an abstraction for the black-box operations to fit inside. This is intended to allow for applications using these interfaces while leveraging existing functionality, which only need to perform data formatting to fit the input defined here. Note, however, some interfaces are defined for certain black-box concepts as part of the SMQTK ecosystem (e.g. in SMQTK-Classifier, SMQTK-Detection, and other SMQTK-* modules).

These interfaces are based on the plugin and configuration features provided by SMQTK-Core, to allow convenient hooks into implementation, discoverability, and factory generation from runtime configuration. This allows for both opaque discovery of interface implementations from a class-method on the interface class object, as well as instantiation of a concrete instance via a JSON-like configuration fed in from an outside resource.

_images/api-docs-fig-01.svg

Figure 1: Abstract Interface Inheritance.

Image Perturbation

Interface: PerturbImage

class nrtk.interfaces.perturb_image.PerturbImage

Algorithm that generates a perturbed image for given input image stimulus as a numpy.ndarray type array.

__call__(image: ndarray[Any, Any], boxes: Iterable[tuple[AxisAlignedBoundingBox, dict[Hashable, float]]] | None = None, **additional_params: Any) → tuple[ndarray[Any, Any], Iterable[tuple[AxisAlignedBoundingBox, dict[Hashable, float]]] | None]

Calls perturb() with the given input image.

__init__() → None

Initializes the PerturbImage.

get_config() → dict[str, Any]

Returns the current configuration of the PerturbImage instance.

Returns:

dict[str, Any]: Configuration dictionary with current settings.

classmethod get_type_string() → str

Returns the fully qualified type string of the PerturbImage class or its subclass.

Returns:

A string representing the fully qualified type, in the format <module>.<class_name>. For example, “my_module.CustomPerturbImage”.

abstract perturb(image: ndarray[Any, Any], boxes: Iterable[tuple[AxisAlignedBoundingBox, dict[Hashable, float]]] | None = None, **additional_params: Any) → tuple[ndarray[Any, Any], Iterable[tuple[AxisAlignedBoundingBox, dict[Hashable, float]]] | None]

Generate a perturbed image for the given image stimulus.

Note perturbers that resize, rotate, or similarly affect the dimensions of an image may impact scoring if bounding boxes are not similarly transformed.

Parameters:

  • image – Input image as a numpy array.

  • boxes – Input bounding boxes as a Iterable of tuples containing bounding boxes. This is the single image output from DetectImageObjects.detect_objects

  • additional_params – Implementation-specific keyword arguments.

Returns:
Perturbed image as numpy array, including matching dtype. Implementations should impart no side

effects upon the input image.

Iterable of tuples containing the bounding boxes for detections in the image. If an implementation

modifies the size of an image, it is expected to modify the bounding boxes as well.

Perturbation Factory

Interface: PerturbImageFactory

class nrtk.interfaces.perturb_image_factory.PerturbImageFactory(perturber: type[PerturbImage], theta_key: str)

Factory class for producing PerturbImage instances of a specified type and configuration.

__getitem__(idx: int) → PerturbImage

Get the perturber for a specific index.

Parameters:

idx – Index of desired perturber.

Raises:

IndexError – The given index does not exist.

Returns:

Perturber corresponding to the given index.

__init__(perturber: type[PerturbImage], theta_key: str) → None

Initialize the factory to produce PerturbImage instances of the given type.

Initialize the factory to produce PerturbImage instances of the given type, varying the given theta_key parameter.

Parameters:

  • perturber – Python implementation type of the PerturbImage interface to produce.

  • theta_key – Perturber parameter to vary between instances.

Raises:

TypeError – Given a perturber instance instead of type.

__iter__() → Iterator[PerturbImage]

Iterator for this factory.

__len__() → int

Number of perturber instances this factory will generate.

__next__() → PerturbImage
Raises:

StopIteration – Iterator exhausted.

Returns:

Next perturber instance.

classmethod from_config(config_dict: dict[str, Any], merge_default: bool = True) → Self

Instantiates a PerturbImageFactory from a configuration dictionary.

Args:

config_dict (dict[str, Any]): Configuration dictionary with parameters for instantiation. merge_default (bool, optional): Whether to merge with default configuration. Defaults to True.

Returns:

An instance of the PerturbImageFactory class.

get_config() → dict[str, Any]

Returns the configuration of the factory instance.

Returns:

dict[str, Any]: Configuration dictionary containing the perturber type and theta_key.

classmethod get_default_config() → dict[str, Any]

Returns the default configuration for the PerturbImageFactory.

This method retrieves a default configuration dictionary, specifying default values for key parameters in the factory. It can be used to create an instance of the factory with preset configurations.

Returns:

dict[str, Any]: A dictionary containing default configuration parameters.

property theta_key: str

Get the perturber parameter to vary between instances.

abstract property thetas: Sequence[Any]

Get the sequence of theta values this factory will iterate over.

Image Metrics

Interface: ImageMetric

class nrtk.interfaces.image_metric.ImageMetric

This interface outlines the computation of a given metric between up to two images.

__call__(img_1: ndarray[Any, Any], img_2: ndarray[Any, Any] | None = None, additional_params: dict[str, Any] | None = None) → float

Calls compute() with the given input image(s) and additional parameters.

Parameters:

  • img_1 – An input image in the shape (height, width, channels).

  • img_2 – An optional input image in the shape (height, width, channels)

  • additional_params – A dictionary containing implementation-specific input param-values pairs.

Returns:

A single scalar value representing an implementation’s computed metric. Implementations should impart no side effects upon either input image or the additional parameters.

abstract compute(img_1: ndarray[Any, Any], img_2: ndarray[Any, Any] | None = None, additional_params: dict[str, Any] | None = None) → float

Given up to two images, and additional parameters, return some given metric about the image(s).

Parameters:

  • img_1 – An input image in the shape (height, width, channels).

  • img_2 – An optional input image in the shape (height, width, channels)

  • additional_params – A dictionary containing implementation-specific input param-values pairs.

Returns:

Returns a single scalar value representing an implementation’s computed metric. Implementations should impart no side effects upon either input image or the additional parameters.

get_config() → dict[str, Any]

Returns the config for the interface.

property name: str

Returns the name of the ImageMetric instance.

This property retrieves the name of the class instance, which can be useful for logging, debugging, or display purposes.

Returns:

str: The name of the ImageMetric instance.

Scoring

Interface: ScoreDetections

class nrtk.interfaces.score_detections.ScoreDetections

Interface abstracting the behavior of taking detections and computing the corresponding metric scores.

Interface abstracting the behavior of taking the actual and predicted detections and computing the corresponding metric scores.

Implementations should verify the validity of the input data.

Note that current implementations are not required to verify nor correct dimension (in)consistency, which may impact scoring.

__call__(actual: Sequence[Sequence[tuple[AxisAlignedBoundingBox, dict[Hashable, Any]]]], predicted: Sequence[Sequence[tuple[AxisAlignedBoundingBox, dict[Hashable, float]]]]) → Sequence[float]

Alias for ScoreDetection.score().

abstract score(actual: Sequence[Sequence[tuple[AxisAlignedBoundingBox, dict[Hashable, Any]]]], predicted: Sequence[Sequence[tuple[AxisAlignedBoundingBox, dict[Hashable, float]]]]) → Sequence[float]

Generate a sequence of scores corresponding to a specific metric.

Parameters:

  • actual – Ground truth bbox and class label pairs.

  • predicted – Output detections from a detector with bbox and class-wise confidence scores.

Returns:

Metric score values as a float-type sequence with the length matching the number of samples in the ground truth input.

End-to-End Generation and Scoring

Interface: GenerateObjectDetectorBlackboxResponse

class nrtk.interfaces.gen_object_detector_blackbox_response.GenerateObjectDetectorBlackboxResponse

This interface describes generation of item-response curves and scores for object detection w.r.t. a blackbox.

This interface describes the generation of item-response curves and scores for object detections with respect to the given black-box object detector after input images are perturbed via the black-box perturber factory. Scoring of these detections is computed with the given black-box scorer.

Note that dimension transformations are not currently accounted for and may impact scoring.

__call__(blackbox_perturber_factories: Sequence[PerturbImageFactory], blackbox_detector: DetectImageObjects, blackbox_scorer: ScoreDetections, img_batch_size: int, verbose: bool = False) → tuple[Sequence[tuple[dict[str, Any], float]], Sequence[Sequence[float]]]

Alias for :meth: .GenerateObjectDetectorBlackboxResponse.generate.

abstract __getitem__(idx: int) → tuple[ndarray[Any, Any], Sequence[tuple[AxisAlignedBoundingBox, dict[Hashable, float]]], dict[str, Any]]

Get the image and ground_truth pair at a particular idx.

generate(blackbox_perturber_factories: Sequence[PerturbImageFactory], blackbox_detector: DetectImageObjects, blackbox_scorer: ScoreDetections, img_batch_size: int, verbose: bool = False) → tuple[Sequence[tuple[dict[str, Any], float]], Sequence[Sequence[float]]]

Generate item-response curves for given parameters.

Parameters:

  • blackbox_perturber_factories – Sequence of factories to perturb stimuli.

  • blackbox_detector – Detector to generate detections for perturbed stimuli.

  • blackbox_scorer – Scorer to score detections.

  • img_batch_size – The number of images to predict and score upon at once.

  • verbose – Increases the verbosity of progress updates.

Returns:

Item-response curve and scores for each input stimuli