Diagnostics

Diagnostics: warnings the framework can raise about its own measures.

This module is what distinguishes the package from a collection of formulas. Each check corresponds to a result about when a measure behaves as a practitioner is likely to assume, and fires when it does not:

  • applying a weighted-averaging interpretation to a measure that is not aggregation-decomposable, where the pooled value may lie outside the range of the per-class values entirely;

  • reporting a tn-dependent measure on one-hot multiclass data, where the attainable floor is bounded away from the nominal minimum by an amount growing with the number of classes;

  • selecting an operating point by maximising a measure whose optimal decision rule is not the Bayes rule for 0/1 loss — degenerately so for recall and specificity;

  • macro-averaging with undefined per-class entries, where the convention chosen can shift the reported value materially;

  • reporting micro-precision, micro-recall or micro-F1 on single-label data, where all three collapse onto accuracy.

class aggmetrics.diagnostics.Finding(code, severity, message, detail)

Bases: object

One diagnostic result.

Parameters:
aggmetrics.diagnostics.diagnose(im, measures=None, *, aggregation='macro', emit_warnings=False)

Run all checks against an IndicatorMatrix.

Returns a list of Finding. With emit_warnings=True each finding of severity 'warning' is also raised as a MeasureWarning, which makes the checks usable inside a test suite or a CI pipeline.

Parameters:
Return type:

List[Finding]

aggmetrics.diagnostics.check_hull(im, measure, aggregation='macro')

Check whether the pooled value lies within the per-class range.

Any weighted average of the per-class values must lie between their minimum and maximum. A micro value outside that interval therefore proves that no weighting reproduces it, so no weighted-averaging interpretation of micro-averaging is available for that measure.

Parameters:
Return type:

Finding | None

aggmetrics.diagnostics.micro_macro_gap(im, measure)

Decompose the micro–macro gap for a decomposable measure.

For an aggregation-decomposable measure the gap equals the covariance between the per-class denominators and the per-class values, normalised by the mean denominator. The identity does not extend beyond that class, and the returned dictionary reports exact=False when it does not apply.

Parameters:

measure (str | BinaryMeasure)

Return type:

dict

aggmetrics.diagnostics.uniform_error_deviation(im)

Compute the mean uniform-error deviation across classes in multiclass OvR.

Proposition 6(2) of the companion paper proves that One-vs-Rest specificity is strictly prior-free if and only if the off-diagonal error rates into class j are uniform ($R_{kj} = text{const}$ for all $k neq j$). This function computes the mean over classes $j$ of the sample standard deviation of $R_{kj}$ over $k neq j$.

Return type:

float

aggmetrics.diagnostics.check_ovr_skew_drift(im, threshold=0.01)

Check whether multiclass One-vs-Rest specificity and balanced accuracy suffer from non-trivial prior drift due to asymmetric off-diagonal error rates.

Parameters:

threshold (float)

Return type:

Finding | None

aggmetrics.diagnostics.check_redundancy(im, measures=None)

Check whether a suite of measures exceeds the intrinsic degrees of freedom.

As shown in Proposition 3 and Remark 2 of the companion paper, a confusion matrix has strictly d = m(m-1) independent degrees of freedom (or d = 2 in binary classification, spanned by the canonical orthogonal basis (recall, specificity)). A benchmark suite with k > d measures inevitably contains collinear gradients and functional redundancy (e.g. reporting accuracy, balanced accuracy, and F1 alongside recall and specificity under fixed priors).

Parameters:

measures (Sequence[str] | None)

Return type:

Finding | None

exception aggmetrics.diagnostics.MeasureWarning

Bases: UserWarning

Raised when a measure is being used outside the range where it behaves as commonly assumed.