Property inference

Deriving a measure’s structural properties from its formula.

The framework’s generative claim is that any function of the four confusion-matrix counts extends to the other settings by substituting an aggregation operator. Taken alone that is cheap: the extensions are always defined. What makes an extension usable is knowing which identities hold for it, and this module shows that those properties need not be supplied by hand — they are recoverable from the formula itself.

Four properties are inferred:

Decomposability. A measure admits the representation \(M = \langle a, v\rangle / \langle b, v\rangle\) iff \(M(v)\,\langle b, v\rangle - \langle a, v\rangle = 0\) for all count vectors \(v\). Sampling count vectors turns this into a homogeneous linear system in the eight unknowns \((a, b)\), solved by taking the right singular vector of smallest singular value. Because a near-null direction can arise by chance, the recovered coefficients are then validated on count vectors that were not used to fit them; only if they reproduce the measure there is the measure reported as decomposable. This both decides the property and returns the coefficients, which is what the Bayes threshold and the weighted-averaging identity need.

Skew-invariance. Tested directly against the definition, by rescaling the positive- and negative-class counts independently.

Dependence on tn at tp = 0. Decides whether the measure can attain its nominal minimum under one-hot multiclass encoding.

Direction. Whether correcting a prediction increases the value.

Inference is numerical and therefore not a proof; verify_inference checks it against the measures whose properties are known analytically, and the test suite requires exact agreement on all of them.

class aggmetrics.inference.InferredProperties(linear, skew_invariant, tn_dependent_at_zero, greater_is_better, residual, diagnostics)

Bases: object

Structural properties recovered from a formula.

Parameters:
aggmetrics.inference.infer_properties(fn, *, seed=0, tol=1e-08)

Recover the structural properties of a binary measure from its formula.

Parameters:

  • fn (Callable[[float, float, float, float], float]) – A callable fn(tp, fp, fn, tn) -> float. It may return NaN on degenerate inputs; such points are skipped.

  • seed (int) – Seed for the count vectors sampled during inference.

  • tol (float) – Relative tolerance for the held-out validation of the linear-fractional fit, and for the invariance tests.

Return type:

InferredProperties

Examples

>>> from aggmetrics.inference import infer_properties
>>> p = infer_properties(lambda tp, fp, fn, tn: tp / (tp + fn))   # recall
>>> p.decomposable, p.skew_invariant
(True, True)
>>> p = infer_properties(lambda tp, fp, fn, tn:
...     0.5 * (tp / (tp + fn) + tn / (tn + fp)))                  # balanced acc.
>>> p.decomposable, p.skew_invariant
(False, True)
aggmetrics.inference.register_measure(name, fn, *, aliases=(), note='', overwrite=False, seed=0, **overrides)

Add a user-supplied measure, inferring what is not given explicitly.

This is the generative claim as an operation: supply a binary formula and the measure becomes available under all four aggregation operators, with the properties that determine which identities apply to it derived automatically. Any property may be overridden by keyword if it is known analytically.

Returns the registered BinaryMeasure.

Examples

>>> import numpy as np
>>> from aggmetrics import IndicatorMatrix, register_measure
>>> # Prevalence-threshold, a measure not in the registry
>>> def prevalence_threshold(tp, fp, fn, tn):
...     r = tp / (tp + fn); s = tn / (tn + fp)
...     d = r + s - 1
...     return (np.sqrt(r * (1 - s)) + s - 1) / d if abs(d) > 1e-12 else np.nan
>>> m = register_measure('prevalence_threshold', prevalence_threshold)
>>> m.decomposable, m.skew_invariant
(False, True)
>>> im = IndicatorMatrix([0, 1, 2, 2, 1, 0], [0, 2, 2, 1, 1, 1],
...                      task='multiclass')
>>> isinstance(im.compute('prevalence_threshold', 'macro'), float)
True
Parameters:
aggmetrics.inference.verify_inference(names=None, *, seed=0)

Compare inferred properties against the registry’s analytic annotations.

Returns a mapping from measure name to a record of agreement. Used by the test suite: inference must reproduce every hand-derived property, which is a check on the inference and on the annotations at once.

Parameters:
Return type:

Dict[str, dict]