Classification Performance Metrics

Accuracy is one number. Classification produces four outcomes. The entire story of model quality — who you're catching, who you're missing, and what false alarms cost — lives in the confusion matrix and the curves derived from it. This post computes every metric by hand on a loan default dataset, then shows when each metric misleads.

Anchor dataset: 20-sample loan default predictions from a logistic regression model.

import numpy as np

y_true = np.array([1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0])
y_prob = np.array([0.95,0.89,0.78,0.71,0.65,0.42,0.38,0.22,
                   0.81,0.63,0.45,0.41,0.35,0.28,0.19,0.15,0.11,0.08,0.05,0.03])
# 8 actual defaults, 12 actual non-defaults

Step 1: Building the Confusion Matrix at Threshold 0.5

Apply threshold 0.5 manually:

• Predicted positive (ŷ=1): y_prob > 0.5 → samples with prob [0.95, 0.89, 0.78, 0.71, 0.65, 0.81, 0.63] = 7 samples
  • True defaults among them: [0.95✓, 0.89✓, 0.78✓, 0.71✓, 0.65✓] = 5 TP
  • True non-defaults: [0.81✗, 0.63✗] = 2 FP
• Predicted negative (ŷ=0): 13 samples
  • True defaults missed: [0.42, 0.38, 0.22] = 3 FN
  • True non-defaults: 10 TN

from sklearn.metrics import confusion_matrix

y_pred = (y_prob >= 0.5).astype(int)
cm = confusion_matrix(y_true, y_pred)
print(cm)

Step 2: All Metrics from the Confusion Matrix

Business interpretation for loan default:

• Precision = 0.714: of 7 loans we flagged as high-risk, 5 were actual defaults. 2 customers were denied loans unnecessarily.
• Recall = 0.625: of 8 actual defaults, we caught 5. We missed 3 defaulters who received loans and likely won't repay them.
• Which matters more? A bank losing money on missed defaults (FN) typically cares more about Recall. A customer discrimination lawsuit from false alarms (FP) shifts priority to Precision. The right metric depends on the asymmetry of the business cost.
• Accuracy = 75% is misleading: if 2% of loans default and you always predict "no default," accuracy = 98%. But recall = 0% — you've detected nothing.

Step 3: The Precision-Recall Tradeoff

As you lower the threshold, you flag more samples as positive (higher recall, lower precision). As you raise it, fewer are flagged (higher precision, lower recall):

At threshold 0.7 and 0.9: precision = 1.0 because the only flagged samples are true positives. But recall drops — we're missing more defaulters. At threshold 0.3: catch 7 of 8 defaulters but also flag 5 non-defaulters.

Step 4: ROC Curve and AUC

The ROC curve plots True Positive Rate (Recall) vs False Positive Rate at each threshold:

from sklearn.metrics import roc_auc_score

auc = roc_auc_score(y_true, y_prob)
print(f"AUC-ROC: {auc:.4f}")

AUC = 0.875 means: if you randomly pick one defaulter and one non-defaulter from the dataset, there's an 87.5% chance the model assigns a higher probability to the defaulter. AUC is threshold-independent — it measures the model's discriminative ability across all possible thresholds.

Step 5: F1 Score and the Beta-F Score

F1 is the harmonic mean of Precision and Recall:

$F_1=\frac{2\times P\times R}{P+R}=\frac{2\times 0.714\times 0.625}{0.714+0.625}=\frac{0.893}{1.339}=0.667$

The harmonic mean is lower than the arithmetic mean ($(0.714+0.625)/2=0.670$) and is dominated by whichever is smaller — a model with Precision=0.99 but Recall=0.10 gets F1=0.18, not a flattering 0.55.

When Recall matters more than Precision (catching defaulters is critical), use $F_{\beta }$ with $\beta >1$:

$F_{\beta }=(1+{\beta }^2)\cdot \frac{P\times R}{{\beta }^2\times P+R}$

$F_2$ (Recall weighted 2× more):

$F_2=\frac{5\times 0.714\times 0.625}{4\times 0.714+0.625}=\frac{2.232}{3.481}=0.641$

$F_{0.5}$ (Precision weighted 2× more):

$F_{0.5}=\frac{1.25\times 0.714\times 0.625}{0.25\times 0.714+0.625}=\frac{0.558}{0.804}=0.694$

$F_2=0.641<F_1=0.667$ because low recall is penalized harder. $F_{0.5}=0.694>F_1$ because the model's precision of 0.714 is respectable.

Code Summary

from sklearn.metrics import classification_report

print(classification_report(y_true, y_pred, target_names=['No Default', 'Default']))

Metric Selection Guide

Related Concepts and Honest Limitations

The confusion matrix and all derived metrics depend on the chosen threshold. A model with AUC-ROC = 0.875 and F1 = 0.667 at threshold 0.5 might have F1 = 0.75 at threshold 0.35. Always visualize the Precision-Recall and ROC curves before committing to a threshold — the threshold should be chosen by the business cost ratio of FP to FN, not arbitrarily set to 0.5.

AUC-ROC = 0.875 looks strong here, but on a severely imbalanced dataset (99% non-default), a model that always outputs a slightly lower probability for the 1% fraud samples can achieve AUC = 0.95 while still being essentially useless for fraud detection. Post 07 covers this with the AUC-PR metric for imbalanced classification.

Test Your Understanding

1. The confusion matrix gives TP=5, FP=2, FN=3, TN=10. If the bank loses $50k per missed defaulter (FN) and $5k per false alarm (FP), what is the total expected cost at threshold 0.5? At threshold 0.3 (where TP=7, FP=5, FN=1, TN=7)?

2. AUC-ROC = 0.875 means an 87.5% chance that a randomly drawn defaulter has a higher predicted probability than a randomly drawn non-defaulter. If you shuffle the predicted probabilities randomly (destroying the model), what would AUC-ROC be?

3. A model achieves Precision=0.90 and Recall=0.30. The F1 is 0.45. A second model has Precision=0.60 and Recall=0.60. Its F1 is also 0.60. Which model does the harmonic mean favor, and why is this the right choice?

4. We computed $F_2=0.641$ and $F_{0.5}=0.694$. As $\beta \to \infty$, what value does $F_{\beta }$ converge to? As $\beta \to 0$?

5. The miss rate (FNR = 0.375) and recall (TPR = 0.625) sum to 1.0. Is this always true? Prove it from the formulas.