Measures

Binary performance measures, each carrying the metadata the framework needs to reason about it.

A measure is not merely a function of the four confusion-matrix counts. Whether the framework’s identities apply to it depends on structural properties that are recorded here alongside the formula:

linear

Coefficients \((\mathbf{a}, \mathbf{b})\) such that \(M = \langle a, v\rangle / \langle b, v\rangle\) with \(v = (tp, fp, fn, tn)\), or None if no such representation exists. A measure with coefficients is aggregation-decomposable: micro-averaging equals denominator-weighted macro-averaging. Without them the identity fails, and the pooled value need not even lie in the convex hull of the per-class values.

skew_invariant

Whether the measure is unchanged when the two classes are rescaled independently, equivalently whether it is a function of recall and specificity alone.

tn_dependent_at_zero

Whether the value at tp == 0 still depends on tn. This determines whether the measure can attain its nominal minimum under one-hot multiclass encoding, where an incorrectly classified example still contributes m - 2 true negatives.

greater_is_better

Direction, needed for ranking and for the scikit-learn scorer interface.

class aggmetrics.measures.BinaryMeasure(name, fn, greater_is_better=True, linear=None, skew_invariant=False, tn_dependent_at_zero=False, aliases=<factory>, note='')

Bases: object

A binary measure together with its structural properties.

Parameters:
property decomposable: bool

True iff micro-averaging equals denominator-weighted macro-averaging.

bayes_threshold()

Optimal decision threshold on the conditional probability, or None.

For a measure whose numerator is linear with coefficients (alpha, -beta, -gamma, delta) and whose denominator does not depend on the predictions, the population-optimal rule is to predict positive when the conditional probability exceeds (beta + delta) / (alpha + beta + gamma + delta).

Returns None when the measure is not of that form, in which case no such closed form exists and an iterative procedure is required.

Return type:

float | None

erm_consistent()

Whether the measure is maximised by the Bayes rule for 0/1 loss.

Return type:

bool | None

attainable_floor(m)

Value attained under one-hot encoding when every prediction is wrong.

With m classes and n examples all misclassified, the pooled counts are tp = 0, fp = fn = n and tn = n (m - 2). Measures that do not depend on tn at tp = 0 reach their nominal minimum; the rest are bounded away from it by an amount that grows with m.

Parameters:

m (int)

Return type:

float

aggmetrics.measures.get_measure(name)

Look up a measure by name or alias.

Parameters:

name (str | BinaryMeasure)

Return type:

BinaryMeasure

aggmetrics.measures.list_measures(decomposable=None, skew_invariant=None)

Canonical measure names, optionally filtered by structural property.

Parameters:

  • decomposable (bool | None)

  • skew_invariant (bool | None)

Return type:

list[str]