Prior shift

Evaluating a measure under a different class prior.

A classifier’s conditional rates, P(predicted k | true j), are what a test set measures. The measures that are not skew-invariant (Definition 11 and Theorem 5 of the companion paper; Proposition 6 for one-vs-rest rates) also depend on how often each class occurs, so the same classifier scores differently where the prevalence differs. under_prior answers “what would this measure be if the classes occurred with these frequencies?” by keeping the rates of the data and replacing the class sizes.

The assumption is the usual one for a change of prior (label shift): the rates of each class do not change, only how many examples it has. It holds when the class-conditional distribution of the inputs is the same in both settings. It does not hold when the shift also changes what the examples look like.

class aggmetrics.shift.PriorShifted(model, priors, task)

Bases: object

The data of an IndicatorMatrix at another prior.

Built by under_prior(). It offers the parts of the IndicatorMatrix interface that depend only on the confusion matrix: compute, accuracy, counts and per_class.

Parameters:

  • model (Rates)

  • task (str)

property n: int

Total number of examples, as in the original data.

per_class()

Per-class (or per-label) (tp, fp, fn, tn), one row per class.

Return type:

ndarray

property counts

Pooled (tp, fp, fn, tn).

compute(measure, aggregation='micro', *, convention='exclude')

Value of measure at the new prior, under the operator aggregation.

Parameters:
Return type:

float

accuracy()

Accuracy, defined as in IndicatorMatrix.accuracy.

Return type:

float

aggmetrics.shift.under_prior(im, priors)

Re-evaluate im as if its classes occurred with the frequencies priors.

Parameters:

  • im – A binary, multiclass or multilabel IndicatorMatrix without a mask or a t-norm.

  • priors – Binary: the prevalence of the positive class, a number in [0, 1]. Multiclass: the frequency of each class, non-negative and summing to 1. Multilabel: the prevalence of each label, one number in [0, 1] per label.

Returns:

An object with compute, accuracy, counts and per_class. At the prior of the data itself it reproduces im exactly.

Return type:

PriorShifted

Examples

>>> import numpy as np
>>> from aggmetrics import IndicatorMatrix, under_prior
>>> y = np.array([1] * 80 + [0] * 120)
>>> p = np.array([1] * 60 + [0] * 20 + [1] * 30 + [0] * 90)
>>> im = IndicatorMatrix(y, p, task="binary")
>>> round(im.compute("precision"), 3)
0.667
>>> shifted = under_prior(im, 0.05)             # 5% positives instead of 40%
>>> round(shifted.compute("precision"), 3)
0.136
>>> round(shifted.compute("recall"), 3) == round(im.compute("recall"), 3)
True