Classification Performance Metrics

Your logistic regression model outputs a probability — 0.95, 0.63, 0.08 — for each loan applicant. At some threshold you convert those to predictions: default or no default. Now you need to know: is the model any good? The answer depends on what you care about. Catching all defaulters (even if you flag some safe applicants) or avoiding false alarms (even if you miss a few defaulters) are fundamentally different goals, and they require different metrics.

One number — accuracy — is never enough. A model that always predicts "no default" scores 99% on a dataset with 1% defaults. The four cells of the confusion matrix — true positives, false positives, true negatives, false negatives — tell the real story. From those four numbers come precision, recall, F1, the ROC curve, and AUC. Each answers a different business question.

This post computes every metric by hand on a 20-sample loan default dataset. You'll see the formula, the substitution, and the business interpretation for each one — and, critically, when each metric can mislead.

What Classification Metrics Tell You

Classification metrics start from the confusion matrix: a 2×2 grid counting correct and incorrect predictions for each class. From that grid: accuracy measures overall correctness (but is useless on imbalanced data), precision measures false alarm cost (of the loans you flagged, how many actually defaulted?), recall measures miss rate (of the actual defaulters, how many did you catch?), and the ROC curve measures discriminative ability across all thresholds. The right metric depends on the asymmetry of the business cost: missing a defaulter and flagging a safe applicant are not equally expensive.

The Plan — Five Steps from Confusion Matrix to Metric Selection

We'll build the confusion matrix at threshold 0.5 and extract all seven standard metrics from it. Then we'll see how those metrics change as the threshold moves, plot the ROC and Precision-Recall curves, and cover the F-beta family for when recall and precision have different importance. Each metric is computed on the same 20-sample dataset.

Anchor dataset: 20-sample loan default predictions from a logistic regression model.

import numpy as np

y_true = np.array([1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0])
y_prob = np.array([0.95,0.89,0.78,0.71,0.65,0.42,0.38,0.22,
                   0.81,0.63,0.45,0.41,0.35,0.28,0.19,0.15,0.11,0.08,0.05,0.03])
# 8 actual defaults, 12 actual non-defaults

Step 1: Building the Confusion Matrix at Threshold 0.5

Apply threshold 0.5 manually:

• Predicted positive (ŷ=1): y_prob > 0.5 → samples with prob [0.95, 0.89, 0.78, 0.71, 0.65, 0.81, 0.63] = 7 samples
  • True defaults among them: [0.95✓, 0.89✓, 0.78✓, 0.71✓, 0.65✓] = 5 TP
  • True non-defaults: [0.81✗, 0.63✗] = 2 FP
• Predicted negative (ŷ=0): 13 samples
  • True defaults missed: [0.42, 0.38, 0.22] = 3 FN
  • True non-defaults: 10 TN

from sklearn.metrics import confusion_matrix

y_pred = (y_prob >= 0.5).astype(int)
cm = confusion_matrix(y_true, y_pred)
print(cm)

[[10  2]
 [ 3  5]]

✓ Step 1 complete. At threshold 0.5, the confusion matrix is TP=5, FP=2, FN=3, TN=10. The model caught 5 of 8 actual defaulters but flagged 2 safe applicants and missed 3 defaulters.

Step 2: All Metrics from the Confusion Matrix

Business interpretation for loan default:

• Precision = 0.714: of 7 loans we flagged as high-risk, 5 were actual defaults. 2 customers were denied loans unnecessarily.
• Recall = 0.625: of 8 actual defaults, we caught 5. We missed 3 defaulters who received loans and likely won't repay them.
• Which matters more? A bank losing money on missed defaults (FN) typically cares more about Recall. A customer discrimination lawsuit from false alarms (FP) shifts priority to Precision. The right metric depends on the asymmetry of the business cost.
• Accuracy = 75% is misleading: if 2% of loans default and you always predict "no default," accuracy = 98%. But recall = 0% — you've detected nothing.

✓ Step 2 complete. Seven metrics computed from the confusion matrix. Precision (0.714) answers "how many flagged loans actually default?" and Recall (0.625) answers "how many defaulters did we catch?" — different business questions, different metrics.

Step 3: The Precision-Recall Tradeoff

As you lower the threshold, you flag more samples as positive (higher recall, lower precision). As you raise it, fewer are flagged (higher precision, lower recall):

At threshold 0.7 and 0.9: precision = 1.0 because the only flagged samples are true positives. But recall drops — we're missing more defaulters. At threshold 0.3: catch 7 of 8 defaulters but also flag 5 non-defaulters.

✓ Step 3 complete. As threshold drops from 0.9 to 0.3, recall rises from 0.25 to 0.875 while precision falls from 1.0 to 0.583. The tradeoff is baked into the data — no free lunch.

Step 4: ROC Curve and AUC

The ROC curve plots True Positive Rate (Recall) vs False Positive Rate at each threshold:

from sklearn.metrics import roc_auc_score

auc = roc_auc_score(y_true, y_prob)
print(f"AUC-ROC: {auc:.4f}")

AUC-ROC: 0.8750

AUC = 0.875 means: if you randomly pick one defaulter and one non-defaulter from the dataset, there's an 87.5% chance the model assigns a higher probability to the defaulter. AUC is threshold-independent — it measures the model's discriminative ability across all possible thresholds.

✓ Step 4 complete. AUC-ROC = 0.875 — an 87.5% chance that a random defaulter is ranked above a random non-defaulter. AUC is threshold-independent, making it useful for comparing models without committing to a cutoff.

Step 5: F1 Score and the Beta-F Score

F1 is the harmonic mean of Precision and Recall:

$F_1=\frac{2\times P\times R}{P+R}=\frac{2\times 0.714\times 0.625}{0.714+0.625}=\frac{0.893}{1.339}=0.667$

The harmonic mean is lower than the arithmetic mean ($(0.714+0.625)/2=0.670$) and is dominated by whichever is smaller — a model with Precision=0.99 but Recall=0.10 gets F1=0.18, not a flattering 0.55.

When Recall matters more than Precision (catching defaulters is critical), use $F_{\beta }$ with $\beta >1$:

$F_{\beta }=(1+{\beta }^2)\cdot \frac{P\times R}{{\beta }^2\times P+R}$

$F_2$ (Recall weighted 2× more):

$F_2=\frac{5\times 0.714\times 0.625}{4\times 0.714+0.625}=\frac{2.232}{3.481}=0.641$

$F_{0.5}$ (Precision weighted 2× more):

$F_{0.5}=\frac{1.25\times 0.714\times 0.625}{0.25\times 0.714+0.625}=\frac{0.558}{0.804}=0.694$

$F_2=0.641<F_1=0.667$ because low recall is penalized harder. $F_{0.5}=0.694>F_1$ because the model's precision of 0.714 is respectable.

As β increases, the curve descends toward the recall floor (0.625). As β → 0, the curve rises toward the precision ceiling (0.714). F₁ sits at the crossing point where precision and recall are weighted equally.

✓ Step 5 complete. F1 = 0.667 is the harmonic mean of precision and recall. F₂ = 0.641 weights recall higher; F₀.₅ = 0.694 weights precision higher. Choose F_β based on which error costs more.

Code Summary

from sklearn.metrics import classification_report

print(classification_report(y_true, y_pred, target_names=['No Default', 'Default']))

precision    recall  f1-score   support

  No Default       0.77      0.83      0.80        12
     Default       0.71      0.62      0.67         8

    accuracy                           0.75        20
   macro avg       0.74      0.73      0.73        20
weighted avg       0.75      0.75      0.74        20

Metric Selection Guide

Related Concepts

The confusion matrix is the foundation for all classification metrics in this series: it's how we evaluate the logistic regression models from Posts 01 and 02. The same TP/FP/FN/TN structure carries into multiclass evaluation (confusion matrices grow to $K\times K$ per-class metrics). The threshold-dependent nature of precision and recall is why AUC-ROC exists — it measures discriminative ability without committing to a cutoff — and why AUC-PR (covered in Post 07) is preferred for imbalanced data.

Honest Limitations

The confusion matrix and all derived metrics depend on the chosen threshold. A model with AUC-ROC = 0.875 and F1 = 0.667 at threshold 0.5 might have F1 = 0.75 at threshold 0.35. Always visualize the Precision-Recall and ROC curves before committing to a threshold — the threshold should be chosen by the business cost ratio of FP to FN, not arbitrarily set to 0.5.

AUC-ROC = 0.875 looks strong here, but on a severely imbalanced dataset (99% non-default), a model that always outputs a slightly lower probability for the 1% fraud samples can achieve AUC = 0.95 while still being essentially useless for fraud detection. Post 07 covers this with the AUC-PR metric for imbalanced classification.

Test Your Understanding

1. The confusion matrix gives TP=5, FP=2, FN=3, TN=10. If the bank loses $50k per missed defaulter (FN) and $5k per false alarm (FP), what is the total expected cost at threshold 0.5? At threshold 0.3 (where TP=7, FP=5, FN=1, TN=7)?

2. AUC-ROC = 0.875 means an 87.5% chance that a randomly drawn defaulter has a higher predicted probability than a randomly drawn non-defaulter. If you shuffle the predicted probabilities randomly (destroying the model), what would AUC-ROC be?

3. A model achieves Precision=0.90 and Recall=0.30. The F1 is 0.45. A second model has Precision=0.60 and Recall=0.60. Its F1 is also 0.60. Which model does the harmonic mean favor, and why is this the right choice?

4. We computed $F_2=0.641$ and $F_{0.5}=0.694$. As $\beta \to \infty$, what value does $F_{\beta }$ converge to? As $\beta \to 0$?

5. The miss rate (FNR = 0.375) and recall (TPR = 0.625) sum to 1.0. Is this always true? Prove it from the formulas.