Pitfall galleryΒΆ
Each notebook starts from a mistake that is easy to make, shows what the usual tools report,
and then what aggmetrics reports. They run in a few seconds, need only NumPy,
scikit-learn, pandas and matplotlib, and are executed in continuous integration.
Notebook |
What it shows |
|---|---|
The floor trap: a classifier wrong on every example can score 0.998 |
A classifier wrong on every example scores 0.998 on label accuracy with 1000 classes |
Micro-F1 equals accuracy; micro-MCC outside the per-class range; the hidden choice for 0/0 |
|
Three models on yeast ranked under 18 measure-operator combinations |
|
Ordinal grades and costs: an error of one grade is not an error of four |
Nine in ten errors are one grade off; a cost matrix changes which prediction is best |
Model selection: the scorer decides, and so does how you average the folds |
The scorer picks the model, pooled and averaged folds differ, and a test that fails on a flawed evaluation |
Long-tailed images: the same model under different class priors |
The same CIFAR-10-LT model scores 0.92 on a long-tailed set and 0.66 on a balanced one; what changes with the class prior |
Recall, specificity and precision are maximised by trivial classifiers; a cost-weighted measure has a closed-form threshold |
|
Soft labels: how much do humans disagree, and does it change the verdict? |
Image classifiers against the distribution of 50 human annotators (CIFAR-10H): does the soft reference change the verdict, where do the models err, and a simulation of when it reverses the verdict |
Cell nuclei (BBBC039): pixel accuracy gives 0.82 to a predictor that outputs nothing; pooled and per-image IoU, and the convention for images with no object |
|
Your own measure: what the package infers, and where it can be wrong |
Register your own formula: what is inferred (decomposable, skew-invariant), what follows on data, and a piecewise measure on which the inference is wrong |
Evaluating framework models: logits, pooling batches instead of averaging a measure over them, accumulating counts, and a Keras callback that logs a measure per epoch |
|
Remark 2 on data: which measures are redundant (rank of the Jacobian), the best pair for a binary problem, and why 63 aggregated measures span only 11 of the 90 directions of a ten-class confusion matrix |