Indicator matrices

The IndicatorMatrix object: a pair of label matrices plus everything derived from them.

Construction accepts the shapes practitioners actually have — class-index vectors, binary indicator matrices, or probability matrices — and converts them once. The four cell matrices are computed lazily and cached, so a full report over many measures costs one pass over the data.

class aggmetrics.core.IndicatorMatrix(y_true, y_pred, task=None, *, threshold=0.5, tnorm=None, mask=None, n_classes=None)

Bases: object

Actual and predicted labels in indicator-matrix form.

Parameters:

  • y_true – Either 1-D arrays of class indices (multiclass, binary, or ordinal) or 2-D binary matrices of shape (n, m) (multilabel, or explicit one-hot). y_pred may also be a real-valued score or probability matrix, in which case threshold or argmax conversion is applied according to task.

  • y_pred – Either 1-D arrays of class indices (multiclass, binary, or ordinal) or 2-D binary matrices of shape (n, m) (multilabel, or explicit one-hot). y_pred may also be a real-valued score or probability matrix, in which case threshold or argmax conversion is applied according to task.

  • task (Optional[Task]) – 'multiclass' enforces one prediction per example via argmax; 'multilabel' thresholds each column independently; 'binary' is multilabel with a single column; 'ordinal' encodes $m$ classes into $m-1$ cumulative threshold indicators. Inferred when omitted.

  • threshold (float) – Threshold for multilabel conversion of scores. Ignored for multiclass.

  • tnorm (Optional[str]) – If given, y_true and y_pred are treated as soft in [0,1] and combined with this t-norm instead of Boolean conjunction. 'product' is the only marginally consistent choice.

  • mask – Optional observation mask; zero entries are treated as unobserved.

  • n_classes (Optional[int])

property cells

Cached (TP, FP, FN, TN) cell matrices.

property counts

Pooled (tp, fp, fn, tn) scalars.

per_class()

Per-class (tp, fp, fn, tn), one row per class or label.

per_example()

Per-example (tp, fp, fn, tn), one row per example.

compute(measure, aggregation='micro', *, convention='exclude')

Value of measure under the chosen aggregation operator.

Parameters:
Return type:

float

parts(measure, aggregation='macro')

Per-class or per-example values, before averaging.

Parameters:
Return type:

ndarray

accuracy()

Standard accuracy.

For one-hot multiclass this is sigma_1(TP) / n, which is not the same as compute('label_accuracy'): the latter is (tp+tn)/(nm) and has a floor of (m-2)/m under one-hot encoding. Keeping them separate avoids a conflation that is easy to make and hard to notice.

Return type:

float

subset_accuracy()

Proportion of examples whose predicted label set matches exactly.

Return type:

float

report(measures=None, aggregations=('micro', 'macro', 'exemplar'), *, convention='exclude', as_frame=True)

Table of measures against aggregation operators.

Returns a pandas.DataFrame when pandas is available and as_frame is True, otherwise a dict of dicts.

Parameters:
property degrees_of_freedom: int

Intrinsic independent degrees of freedom of the confusion matrix.

For binary classification ($m=2$), this is $2(1) = 2$, spanned by the canonical orthogonal basis $(text{recall}, text{specificity})$. For $m$-class multiclass classification, this is $m(m-1)$, spanned by the directional off-diagonal conditional misclassification rates $R_{jk}$ ($k neq j$). For multilabel classification with $m$ labels, each label has 2 degrees of freedom ($2m$).

canonical_coordinates()

Extract the canonical orthogonal coordinates (conditional error rates).

For multiclass problems, returns the $m(m-1)$ off-diagonal conditional error rates $R_{jk} = P(hat{Y}=k mid Y=j)$ for $k neq j$, which form an orthogonal basis in Frobenius inner product. For binary problems, returns $(text{recall}, text{specificity})$.

Return type:

ndarray

cost(C, aggregation='micro')

Total or averaged cost under an m x m cost matrix.

Cost = sigma_1((Y C) o Yhat). With C[j,k] = |j-k| this is n times the mean absolute error, and with C[j,k] = (j-k)^2 it is n times the mean squared error.

Parameters:

aggregation (Literal['micro', 'macro', 'weighted', 'exemplar'])

Return type:

float

dominates(other, strict=False)

Check whether this classifier dominates another on all monotonic measures.

By Section 7.11 of the companion paper (which rests on Proposition 3), a classifier A dominates B across all measures respecting confusion-matrix monotonicity if and only if A dominates B elementwise in the canonical coordinate space:

  • In binary: TP_A >= TP_B, FP_A <= FP_B, FN_A <= FN_B, TN_A >= TN_B

  • In multiclass: R_{jk}^{(A)} <= R_{jk}^{(B)} for all off-diagonal errors (k != j) and R_{jj}^{(A)} >= R_{jj}^{(B)} for all on-diagonal recalls.

Parameters:
Return type:

bool

aggmetrics.core.pareto_dominance(im_a, im_b)

Determine Pareto dominance between two classifiers.

Returns:

  • 'A_dominates_B' if A strictly dominates B across all monotonic measures;

  • 'B_dominates_A' if B strictly dominates A;

  • 'equivalent' if both confusion profiles are identical;

  • 'incomparable' if neither dominates the other (the ranking depends on the choice of measure).

Parameters:
Return type:

str