Metric (pymia.evaluation.metric
package)¶
The metric package provides metrics for evaluation of image segmentation, image reconstruction, and regression.
All metrics implement the pymia.evaluation.metric.base.Metric
interface, and can be used with the
pymia.evaluation.evaluator
package to evaluate results
(e.g., with the pymia.evaluation.evaluator.SegmentationEvaluator
).
To implement your own metric and use it with the pymia.evaluation.evaluator.Evaluator
, you need to inherit from
pymia.evaluation.metric.base.Metric
, pymia.evaluation.metric.base.ConfusionMatrixMetric
,
pymia.evaluation.metric.base.DistanceMetric
, pymia.evaluation.metric.base.NumpyArrayMetric
, or
pymia.evaluation.metric.base.SpacingMetric
and implement pymia.evaluation.metric.base.Metric.calculate()
.
Note
The segmentation metrics are selected based on the paper by Taha and Hanbury. We recommend to refer to the paper for guidelines on how to select appropriate metrics, descriptions, and the math.
Taha, A. A., & Hanbury, A. (2015). Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging, 15. https://doi.org/10.1186/s12880-015-0068-x
Base (pymia.evaluation.metric.base
) module¶
The base module provides metric base classes.
- class pymia.evaluation.metric.base.ConfusionMatrix(prediction: numpy.ndarray, reference: numpy.ndarray)[source]¶
Bases:
object
Represents a confusion matrix (or error matrix).
- Parameters
prediction (np.ndarray) – The prediction binary array.
reference (np.ndarray) – The reference binary array.
- class pymia.evaluation.metric.base.ConfusionMatrixMetric(metric: str = 'ConfusionMatrixMetric')[source]¶
Bases:
pymia.evaluation.metric.base.Metric
,abc.ABC
Represents a metric based on the confusion matrix.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.base.DistanceMetric(metric: str = 'DistanceMetric')[source]¶
Bases:
pymia.evaluation.metric.base.Metric
,abc.ABC
Represents a metric based on distances.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.base.Distances(prediction: numpy.ndarray, reference: numpy.ndarray, spacing: tuple)[source]¶
Bases:
object
Represents distances for distance metrics.
- Parameters
prediction (np.ndarray) – The prediction binary array.
reference (np.ndarray) – The reference binary array.
spacing (tuple) – The spacing in mm of each dimension.
See also
Nikolov, S., Blackwell, S., Mendes, R., De Fauw, J., Meyer, C., Hughes, C., … Ronneberger, O. (2018). Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy. http://arxiv.org/abs/1809.04430
- class pymia.evaluation.metric.base.Information(column_name: str, value: str)[source]¶
Bases:
pymia.evaluation.metric.base.Metric
Represents an information “metric”.
Can be used to add an additional column of information to an evaluator.
- Parameters
column_name (str) – The identification string of the information.
value (str) – The information.
- class pymia.evaluation.metric.base.Metric(metric: str = 'Metric')[source]¶
Bases:
abc.ABC
Metric base class.
- Parameters
metric (str) – The identification string of the metric.
- exception pymia.evaluation.metric.base.NotComputableMetricWarning[source]¶
Bases:
RuntimeWarning
Warning class to raise if a metric cannot be computed.
- class pymia.evaluation.metric.base.NumpyArrayMetric(metric: str = 'NumpyArrayMetric')[source]¶
Bases:
pymia.evaluation.metric.base.Metric
,abc.ABC
Represents a metric based on numpy arrays.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.base.SpacingMetric(metric: str = 'SpacingMetric')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
,abc.ABC
Represents a metric based on images with a physical spacing.
- Parameters
metric (str) – The identification string of the metric.
Metric (pymia.evaluation.metric.metric
) module¶
The metric module provides a set of metrics.
- pymia.evaluation.metric.metric.get_classical_metrics()[source]¶
Gets a list of classical metrics.
- Returns
A list of metrics.
- Return type
list[Metric]
- pymia.evaluation.metric.metric.get_distance_metrics()[source]¶
Gets a list of distance-based metrics.
- Returns
A list of metrics.
- Return type
list[Metric]
- pymia.evaluation.metric.metric.get_overlap_metrics()[source]¶
Gets a list of overlap-based metrics.
- Returns
A list of metrics.
- Return type
list[Metric]
- pymia.evaluation.metric.metric.get_reconstruction_metrics()[source]¶
Gets a list with reconstruction metrics.
- Returns
A list of metrics.
- Return type
list[Metric]
Categorical metrics (pymia.evaluation.metric.categorical
) module¶
The categorical module provides metrics to measure image segmentation performance.
- class pymia.evaluation.metric.categorical.Accuracy(metric: str = 'ACURCY')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents an accuracy metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.AdjustedRandIndex(metric: str = 'ADJRIND')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents an adjusted rand index metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.AreaMetric(metric: str = 'AREA')[source]¶
Bases:
pymia.evaluation.metric.base.SpacingMetric
,abc.ABC
Represents an area metric base class.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.AreaUnderCurve(metric: str = 'AUC')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents an area under the curve metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.AverageDistance(metric: str = 'AVGDIST')[source]¶
Bases:
pymia.evaluation.metric.base.SpacingMetric
Represents an average (Hausdorff) distance metric.
Calculates the distance between the set of non-zero pixels of two images using the following equation:
where
is the directed Hausdorff distance and and are the set of non-zero pixels in the images.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.CohenKappaCoefficient(metric: str = 'KAPPA')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a Cohen’s kappa coefficient metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.DiceCoefficient(metric: str = 'DICE')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a Dice coefficient metric with empty target handling, defined as:
where is the prediction and the target.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.FMeasure(beta: float = 1.0, metric: str = 'FMEASR')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a F-measure metric.
- Parameters
beta (float) – The beta to trade-off precision and recall. Use 0.5 or 2 to calculate the F0.5 and F2 measure, respectively.
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.Fallout(metric: str = 'FALLOUT')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a fallout (false positive rate) metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.FalseNegative(metric: str = 'FN')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a false negative metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.FalseNegativeRate(metric: str = 'FNR')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a false negative rate metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.FalsePositive(metric: str = 'FP')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a false positive metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.GlobalConsistencyError(metric: str = 'GCOERR')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a global consistency error metric.
Implementation based on Martin 2001. todo(fabianbalsiger): add entire reference
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.HausdorffDistance(percentile: float = 100.0, metric: str = 'HDRFDST')[source]¶
Bases:
pymia.evaluation.metric.base.DistanceMetric
Represents a Hausdorff distance metric.
Calculates the distance between the set of non-zero pixels of two images using the following equation:
where
is the directed Hausdorff distance and and are the set of non-zero pixels in the images.
- Parameters
percentile (float) – The percentile (0, 100] to compute, i.e. 100 computes the Hausdorff distance and 95 computes the 95th Hausdorff distance.
metric (str) – The identification string of the metric.
See also
Nikolov, S., Blackwell, S., Mendes, R., De Fauw, J., Meyer, C., Hughes, C., … Ronneberger, O. (2018). Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy. http://arxiv.org/abs/1809.04430
- class pymia.evaluation.metric.categorical.InterclassCorrelation(metric: str = 'ICCORR')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents an interclass correlation metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.JaccardCoefficient(metric: str = 'JACRD')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a Jaccard coefficient metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.MahalanobisDistance(metric: str = 'MAHLNBS')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents a Mahalanobis distance metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.MutualInformation(metric: str = 'MUTINF')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a mutual information metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.Precision(metric: str = 'PRCISON')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a precision metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.PredictionArea(slice_number: int = - 1, metric: str = 'PREDAREA')[source]¶
Bases:
pymia.evaluation.metric.categorical.AreaMetric
Represents a prediction area metric.
- Parameters
slice_number (int) – The slice number to calculate the area. Defaults to -1, which will calculate the area on the intermediate slice.
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.PredictionVolume(metric: str = 'PREDVOL')[source]¶
Bases:
pymia.evaluation.metric.categorical.VolumeMetric
Represents a prediction volume metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.ProbabilisticDistance(metric: str = 'PROBDST')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents a probabilistic distance metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.RandIndex(metric: str = 'RNDIND')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a rand index metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.ReferenceArea(slice_number: int = - 1, metric: str = 'REFAREA')[source]¶
Bases:
pymia.evaluation.metric.categorical.AreaMetric
Represents a reference area metric.
- Parameters
slice_number (int) – The slice number to calculate the area. Defaults to -1, which will calculate the area on the intermediate slice.
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.ReferenceVolume(metric: str = 'REFVOL')[source]¶
Bases:
pymia.evaluation.metric.categorical.VolumeMetric
Represents a reference volume metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.Sensitivity(metric: str = 'SNSVTY')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a sensitivity (true positive rate or recall) metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.Specificity(metric: str = 'SPCFTY')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a specificity metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.SurfaceDiceOverlap(tolerance: float = 1, metric: str = 'SURFDICE')[source]¶
Bases:
pymia.evaluation.metric.base.DistanceMetric
Represents a surface Dice coefficient overlap metric.
- Parameters
tolerance (float) – The tolerance of the surface distance in mm.
metric (str) – The identification string of the metric.
See also
Nikolov, S., Blackwell, S., Mendes, R., De Fauw, J., Meyer, C., Hughes, C., … Ronneberger, O. (2018). Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy. http://arxiv.org/abs/1809.04430
- class pymia.evaluation.metric.categorical.SurfaceOverlap(tolerance: float = 1.0, prediction_to_reference: bool = True, metric: str = 'SURFOVLP')[source]¶
Bases:
pymia.evaluation.metric.base.DistanceMetric
Represents a surface overlap metric.
Computes the overlap of the reference surface with the predicted surface and vice versa allowing a specified tolerance (maximum surface-to-surface distance that is regarded as overlapping). The overlapping fraction is computed by correctly taking the area of each surface element into account.
- Parameters
tolerance (float) – The tolerance of the surface distance in mm.
prediction_to_reference (bool) – Computes the prediction to reference if True, otherwise the reference to prediction.
metric (str) – The identification string of the metric.
See also
Nikolov, S., Blackwell, S., Mendes, R., De Fauw, J., Meyer, C., Hughes, C., … Ronneberger, O. (2018). Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy. http://arxiv.org/abs/1809.04430
- class pymia.evaluation.metric.categorical.TrueNegative(metric: str = 'TN')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a true negative metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.TruePositive(metric: str = 'TP')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a true positive metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.VariationOfInformation(metric: str = 'VARINFO')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a variation of information metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.VolumeMetric(metric: str = 'VOL')[source]¶
Bases:
pymia.evaluation.metric.base.SpacingMetric
,abc.ABC
Represents a volume metric base class.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.categorical.VolumeSimilarity(metric: str = 'VOLSMTY')[source]¶
Bases:
pymia.evaluation.metric.base.ConfusionMatrixMetric
Represents a volume similarity metric.
- Parameters
metric (str) – The identification string of the metric.
Continuous metrics (pymia.evaluation.metric.continuous
) module¶
The continuous module provides metrics to measure image reconstruction and regression performance.
- class pymia.evaluation.metric.continuous.CoefficientOfDetermination(metric: str = 'R2')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents a coefficient of determination (R^2) error metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.continuous.MeanAbsoluteError(metric: str = 'MAE')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents a mean absolute error metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.continuous.MeanSquaredError(metric: str = 'MSE')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents a mean squared error metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.continuous.NormalizedRootMeanSquaredError(metric: str = 'NRMSE')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents a normalized root mean squared error metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.continuous.PeakSignalToNoiseRatio(metric: str = 'PSNR')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents a peak signal to noise ratio metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.continuous.RootMeanSquaredError(metric: str = 'RMSE')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents a root mean squared error metric.
- Parameters
metric (str) – The identification string of the metric.
- class pymia.evaluation.metric.continuous.StructuralSimilarityIndexMeasure(metric: str = 'SSIM')[source]¶
Bases:
pymia.evaluation.metric.base.NumpyArrayMetric
Represents a structural similarity index measure metric.
- Parameters
metric (str) – The identification string of the metric.