Your own measure: what the package infers, and where it can be wrong

The 21 measures of the registry cover the usual ones, but a project often has its own: a score with unequal costs, a measure from a clinical guideline, a variant found in a paper. Writing it as a function of the four counts tp, fp, fn, tn is enough for aggmetrics to give it all four aggregation operators. What the measure then is, for aggregation purposes (whether micro equals a weighted macro, whether it survives a change of class balance, whether it depends on tn), is not something the user declares. The package infers it from the formula.

This notebook registers three measures, shows what is inferred and what follows from it, then builds a measure on which the inference is wrong and shows how to detect that and how to correct it.

# Installs the package only where it is missing (for example in Colab).
import importlib.util
if importlib.util.find_spec("aggmetrics") is None:
    %pip install -q aggmetrics scikit-learn matplotlib pandas

import io
import urllib.request
import warnings
warnings.filterwarnings("ignore")
from pathlib import Path

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import KFold, cross_val_predict
from sklearn.multioutput import MultiOutputClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

from aggmetrics import IndicatorMatrix, get_measure, list_measures, register_measure
from aggmetrics.inference import infer_properties, verify_inference

builtin = list_measures()                      # the 21 measures of the registry, before adding any

Three formulas

• cost_accuracy: accuracy in which a missed positive costs five times a false alarm.

• prevalence_threshold: the prevalence below which a positive result is more likely to be a false positive than a true one. Lower is better.

• root_recall: the square root of recall, a monotone transform of a registry measure.

infer_properties takes the formula alone and returns what it recovered. It fits a linear-fractional form to the formula where one exists, and tests invariance to class balance and dependence on tn on sampled counts.

def cost_accuracy(tp, fp, fn, tn):
    # accuracy-like score in which a missed positive costs five times a false alarm
    return 1 - (5 * fn + fp) / (tp + fp + fn + tn)

def prevalence_threshold(tp, fp, fn, tn):
    # the prevalence below which a positive result is more likely false than true (lower is better)
    r, s = tp / (tp + fn), tn / (tn + fp)
    d = r + s - 1
    return (np.sqrt(r * (1 - s)) + s - 1) / d if abs(d) > 1e-12 else np.nan

def root_recall(tp, fp, fn, tn):
    return np.sqrt(tp / (tp + fn))

candidates = {"cost_accuracy": cost_accuracy, "prevalence_threshold": prevalence_threshold, "root_recall": root_recall}
rows = {}
for name, fn in candidates.items():
    p = infer_properties(fn)
    rows[name] = {
        "linear-fractional": p.linear is not None,
        "skew-invariant": p.skew_invariant,
        "tn-dependent at tp = 0": p.tn_dependent_at_zero,
        "higher is better": p.greater_is_better,
    }
pd.DataFrame(rows).T

	linear-fractional	skew-invariant	tn-dependent at tp = 0	higher is better
cost_accuracy	True	False	True	True
prevalence_threshold	False	True	False	False
root_recall	False	True	False	True

Only cost_accuracy is linear-fractional, and that is what makes a measure decomposable: it can be written as a ratio of two sums over the cells, so that pooling cells and averaging per label are tied by an identity. root_recall is a monotone function of recall and still is not decomposable, since the square root breaks the ratio of sums. register_measure runs the same inference and stores the result.

for name, fn in candidates.items():
    m = register_measure(name, fn)
    print(f"{name:22s} decomposable: {str(m.decomposable):5s}  skew-invariant: {str(m.skew_invariant):5s}")

cost_accuracy          decomposable: True   skew-invariant: False
prevalence_threshold   decomposable: False  skew-invariant: True 
root_recall            decomposable: False  skew-invariant: True 

What the properties mean on data

The multilabel benchmark yeast (14 labels) with a logistic regression and a random forest, predicted by 5-fold cross-validation. The table gives the new measures, and recall for reference, under three operators.

URL = "https://raw.githubusercontent.com/rcprati/aggmetrics/main/examples/data/yeast.npz"
local = Path("../data/yeast.npz")
source = local if local.exists() else io.BytesIO(urllib.request.urlopen(URL, timeout=60).read())
with np.load(source) as z:
    X, Y = z["X"], z["Y"].astype(int)

models = {
    "Logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=500)),
    "Random forest": RandomForestClassifier(n_estimators=100, random_state=0, n_jobs=-1),
}
cv = KFold(5, shuffle=True, random_state=0)
ims = {name: IndicatorMatrix(Y, cross_val_predict(MultiOutputClassifier(model), X, Y, cv=cv), task="multilabel")
       for name, model in models.items()}
print("examples, labels:", Y.shape)

examples, labels: (2417, 14)

rows = {}
for name, im in ims.items():
    for op in ("micro", "macro", "weighted"):
        rows[(name, op)] = {ms: im.compute(ms, op) for ms in ("cost_accuracy", "prevalence_threshold", "root_recall", "recall")}
grid = pd.DataFrame(rows).T
grid.round(4)

		cost_accuracy	prevalence_threshold	root_recall	recall
Logistic regression	micro	0.2785	0.3098	0.7595	0.5768
	macro	0.2785	0.3634	0.5343	0.3623
	weighted	0.2402	0.4105	0.7175	0.5768
Random forest	micro	0.2757	0.2780	0.7481	0.5596
	macro	0.2757	0.2661	0.4653	0.3244
	weighted	0.2771	0.3455	0.6911	0.5596

cost_accuracy has the same value under micro and macro (0.2785 and 0.2785 for the logistic regression), and a different one under weighted. That is not a coincidence: each label has the same denominator, n, so the denominator-weighted macro, which equals the micro value for a decomposable measure, weights all labels equally. For recall the denominator is the number of positives, so micro equals weighted (0.5768 and 0.5768) and differs from macro.

The operator can also change the verdict, and the new measures show it with no extra work.

for ms in ("cost_accuracy", "prevalence_threshold"):
    lower = ms == "prevalence_threshold"
    for op in ("micro", "macro", "weighted"):
        a, b = (grid.loc[(n, op), ms] for n in models)
        best = min(models, key=lambda n: grid.loc[(n, op), ms] * (1 if lower else -1))
        print(f"{ms:22s} {op:9s} best: {best:20s} difference {abs(a - b):.4f}")

cost_accuracy          micro     best: Logistic regression  difference 0.0028
cost_accuracy          macro     best: Logistic regression  difference 0.0028
cost_accuracy          weighted  best: Random forest        difference 0.0369
prevalence_threshold   micro     best: Random forest        difference 0.0318
prevalence_threshold   macro     best: Random forest        difference 0.0973
prevalence_threshold   weighted  best: Random forest        difference 0.0649

For cost_accuracy the logistic regression is ahead of the random forest under micro and macro, by 0.0028, and behind it under weighted, by 0.0369. This is one cross-validated run and the 0.0028 was not tested for significance, so the first two rows are close to a tie, but the third is not. For prevalence_threshold (lower is better) the random forest is ahead under all three operators, by 0.03 to 0.10.

The identity, checked

The property that was inferred is used. For a decomposable measure the micro value equals the macro value weighted by the denominator of the measure in each label, and the code below computes that weighted macro directly from the per-label counts.

def per_label_counts(im):
    Yb, Pb = im.Y.astype(bool), im.P.astype(bool)
    return [((Yb & Pb)[:, j].sum(), (~Yb & Pb)[:, j].sum(), (Yb & ~Pb)[:, j].sum(), (~Yb & ~Pb)[:, j].sum())
            for j in range(Yb.shape[1])]

def denominator_weighted_macro(im, name):
    m = get_measure(name)
    counts = per_label_counts(im)
    w = np.array([m.denominator(c) for c in counts])
    v = np.array([m.fn(*c) for c in counts])
    return float((w * v).sum() / w.sum())

im = ims["Random forest"]
rows = {}
for ms in ("cost_accuracy", "recall", "prevalence_threshold", "root_recall"):
    rows[ms] = {
        "micro": im.compute(ms, "micro"),
        "macro": im.compute(ms, "macro"),
        "weighted": im.compute(ms, "weighted"),
        "denominator-weighted macro": denominator_weighted_macro(im, ms) if get_measure(ms).decomposable else np.nan,
        "decomposable": get_measure(ms).decomposable,
    }
pd.DataFrame(rows).T.round(6)

	micro	macro	weighted	denominator-weighted macro	decomposable
cost_accuracy	0.275726	0.275726	0.277088	0.275726	True
recall	0.559613	0.324406	0.559613	0.559613	True
prevalence_threshold	0.278012	0.26614	0.345529	NaN	False
root_recall	0.748073	0.465339	0.691102	NaN	False

For cost_accuracy and recall the two columns agree to six digits. For prevalence_threshold and root_recall the package makes no such claim and the operators give three different values, as the table shows.

When the inference is wrong

The inference looks at the formula through sampled count vectors. A measure that behaves differently outside what is sampled can be misclassified. The measure below is recall that is halved when there are fewer than 2000 positives.

def kinked_recall(tp, fp, fn, tn):
    # recall, except that it is halved when there are fewer than 2000 positives
    return tp / (tp + fn) if tp + fn >= 2000 else 0.5 * tp / (tp + fn)

for seed in range(5):
    p = infer_properties(kinked_recall, seed=seed)
    print("seed", seed, "| linear-fractional:", p.linear is not None, "| skew-invariant:", p.skew_invariant)

seed 0 | linear-fractional: True | skew-invariant: True
seed 1 | linear-fractional: True | skew-invariant: True
seed 2 | linear-fractional: True | skew-invariant: True
seed 3 | linear-fractional: True | skew-invariant: True
seed 4 | linear-fractional: True | skew-invariant: True

Whatever the seed, the inference concludes that the measure is linear-fractional and skew-invariant, like recall. On the counts it samples it behaves as one formula. The consequence appears on data that has both a large label and a small one.

rng = np.random.default_rng(0)
n = 6000
Yk = np.column_stack([rng.random(n) < 0.50, rng.random(n) < 0.02]).astype(int)       # 3000 and about 120 positives
Pk = np.where(rng.random((n, 2)) < 0.8, Yk, 1 - Yk)                                    # each entry right with probability 0.8
imk = IndicatorMatrix(Yk, Pk, task="multilabel")
print("positives per label:", Yk.sum(axis=0))

m = register_measure("kinked_recall", kinked_recall)
print("declared decomposable:", m.decomposable)
row = {
    "micro": imk.compute("kinked_recall", "micro"),
    "denominator-weighted macro": denominator_weighted_macro(imk, "kinked_recall"),
}
print({k: round(v, 4) for k, v in row.items()}, "| gap", round(abs(row["micro"] - row["denominator-weighted macro"]), 4))

positives per label: [3006  109]
declared decomposable: True
{'micro': 0.8048, 'denominator-weighted macro': 0.7904} | gap 0.0144

With 3006 and 109 positives in the two labels, the value for the pooled cells is 0.8048 and the denominator-weighted macro is 0.7904. The identity that the package attached to the measure does not hold, because the pooled cells have more than 2000 positives and the small label alone does not. The measure is not decomposable, and neither is it skew-invariant: changing the prevalence changes which side of 2000 the count falls on.

Any inferred property can be overridden by keyword when the user knows better.

m = register_measure("kinked_recall", kinked_recall, overwrite=True, linear=None, skew_invariant=False)
print("after the override, decomposable:", m.decomposable, "| skew-invariant:", m.skew_invariant)

after the override, decomposable: False | skew-invariant: False

Checking the inference itself

verify_inference compares the inferred properties of registered measures with the analytic ones. For the 21 measures of the registry they agree in every case, which checks the inference and the annotations at once. For a measure with an override, the same report shows the departure.

report = verify_inference(builtin)
print("built-in measures checked:", len(report))
print("inferred properties agree with the analytic ones:", sum(r["agrees"] for r in report.values()))

# for a measure with an override, the same report shows where the declaration departs from the inference
r = verify_inference(["kinked_recall"])["kinked_recall"]
print("kinked_recall | declared decomposable:", r["decomposable_declared"], "| inferred:", r["decomposable_inferred"])

built-in measures checked: 21
inferred properties agree with the analytic ones: 21
kinked_recall | declared decomposable: False | inferred: True

Summary

• A measure given as a function of tp, fp, fn, tn gets all four operators, and the properties that decide which identities hold are inferred. Here that gave micro equal to macro for cost_accuracy, because every label has the same denominator, and no identity for the two non-decomposable measures.

• A new measure can change the verdict with the operator, like the registry measures. Report the operator with the measure.

• Inference works on sampled counts. For a piecewise measure, such as one with a threshold on a count, it can conclude the wrong thing and the identity it attaches then fails on data. Declare such properties by hand with the override keywords, and use verify_inference to see where a declaration and the inference differ.