Handling Imbalanced Datasets

Imagine you just deployed a fraud detection model. Your manager calls you into their office, grinning. "Ninety-nine percent accuracy on the validation set — great work." You nod, accept the praise, and walk back to your desk feeling proud.

Then you review the production logs. In three months, your model has flagged exactly zero transactions as fraudulent. The fraud team caught a dozen cases through manual review that your model greenlit without a second thought. The model had 99% accuracy because 99 out of every 100 transactions are legitimate — predicting "legitimate" every single time gives you 99% accuracy and exactly zero value. Your model is useless, but the metrics say it's perfect.

That sinking feeling — realizing your metrics lied to you — is the real cost of class imbalance. It doesn't just break your accuracy score. It silently biases your model toward the majority class, and by the time you discover the problem, you've already wasted weeks trusting a model that learned nothing. In this post, you'll see exactly what imbalance does to a model, why standard metrics hide the damage, and the evaluation protocol that surfaces the real performance.

What Class Imbalance Actually Means

Class imbalance means one category of your target variable has far fewer samples than the other. This is NOT the same as having a hard classification problem — you can have cleanly separable classes that are still imbalanced. The problem is that standard metrics (like accuracy) measure the wrong thing when classes are lopsided, and models optimize for majority-class performance by default.

The Anchor

Credit card fraud detection — 20 transactions with three features and a binary label. Eighteen are legitimate, two are fraudulent. The two fraudulent transactions are the high-amount, foreign, late-night ones.

import pandas as pd
import numpy as np

data = {
    'amount':     [120, 45, 980, 300, 55, 1200, 75, 430, 22, 600,
                   800, 150, 35, 90, 500, 220, 70, 410, 9500, 4200],
    'hour':       [10, 14, 2,  16, 9,  3,  11, 15, 8,  1,
                   13, 6,  18, 20, 4,  12, 7,  17, 3,  2],
    'is_foreign': [0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
                   0,  0,  0,  0,  0,  0,  0,  0,  1,  1],
    'fraud':      [0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
                   0,  0,  0,  0,  0,  0,  0,  0,  1,  1],
}
df = pd.DataFrame(data)
# Class distribution: 18 legitimate (90%) vs 2 fraudulent (10%)

Every calculation, SVG, and code block in this post uses these exact 20 rows.

The Plan — Four Steps to Understanding Imbalance

We'll walk through four steps. First we see how a naive model achieves 90% accuracy while catching zero fraud, then examine why each standard metric lies on its own, then train a real classifier and watch it fail, and finally set up a proper evaluation protocol.

The Baseline Trap

Consider the simplest possible model: predict "legitimate" for every transaction, no matter what.

y_true = df['fraud'].values
y_pred_always_0 = np.zeros(20, dtype=int)

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

accuracy_score(y_true, y_pred_always_0)

0.9

Ninety percent accuracy. Two of the 20 transactions are fraudulent, and the model catches zero of them. Accuracy hides the failure completely.

Compute the four metrics that tell the full story:

accuracy  = accuracy_score(y_true, y_pred_always_0)
precision = precision_score(y_true, y_pred_always_0, pos_label=1, zero_division=0)
recall    = recall_score(y_true, y_pred_always_0, pos_label=1)
f1        = f1_score(y_true, y_pred_always_0, pos_label=1)
print(f"Accuracy:  {accuracy:.2f}")
print(f"Precision: {precision:.2f}")
print(f"Recall:    {recall:.2f}")
print(f"F1:        {f1:.2f}")

Accuracy:  0.90
Precision: 0.00
Recall:    0.00
F1:        0.00

Precision is 0.0 (no fraud predicted), recall is 0.0 (every fraud missed), F1 collapses to 0. The 0.90 accuracy number is the lie. Three of the four metrics say the model is useless; accuracy alone says it works.

Visually, the class distribution and the baseline confusion matrix tell the same story in different shapes:

Eighteen true negatives, two false negatives. Every fraud row sits in the FN cell.

✓ Step 1 complete. The always-0 baseline scored 90% accuracy but caught zero fraud. Precision (0.0), recall (0.0), and F1 (0.0) exposed the failure that accuracy hid.

Why Standard Metrics Lie

Accuracy, precision, and recall each fail in a specific way on imbalanced data:

• Accuracy is dominated by the majority class. With 18 of 20 samples negative, predicting "0" everywhere is already 90% accurate.
• Precision alone is 1.0 by default when the model never predicts positive. "Of all my fraud predictions, 100% were correct" — vacuously true when the model makes zero fraud predictions.
• Recall alone is 1.0 when the model flags everything as positive. "I caught all the fraud" — at the cost of a false alarm on every legitimate transaction.
• F1 is the harmonic mean of precision and recall. It collapses to 0 when either is 0, which is exactly what an always-0 baseline gets.
• Balanced accuracy averages TPR and TNR. For the always-0 baseline: (0/2 + 18/18) / 2 = 0.50. A random classifier also scores 0.50, so balanced accuracy exposes the trivial prediction.
• PR-AUC (area under the precision-recall curve) is near 0 for a useless classifier. Unlike ROC-AUC, it does not reward the model for ranking a flood of negatives above each other.
• ROC-AUC can be misleadingly high on imbalanced data. A model that barely moves the threshold can score 0.85 on a 99:1 dataset because the true negative axis stretches far to the right.

This is NOT a critique of accuracy as a metric — accuracy is fine for balanced datasets. The problem is using the wrong tool for an imbalanced job.

Let's trace each metric through the formulas using our actual numbers — you'll see exactly why accuracy paints the wrong picture:

Here are the five metrics computed on the always-0 baseline. The formula column shows the arithmetic with our anchor values substituted in:

A model that only moves accuracy should not be trusted. The fix is to report precision, recall, F1, balanced accuracy, and PR-AUC on the positive class.

✓ Step 2 complete. We now know which metrics to trust — F1, balanced accuracy, and PR-AUC — and which to distrust (accuracy and ROC-AUC on their own).

Naive Classifier on the Anchor

A standard LogisticRegression() trained on the same 20 rows, predicting on the same 20 rows, shows what happens when the loss function weights every sample equally:

I set random_state=42 so the random initialization of the optimizer is reproducible. Change or remove it to see how the solution varies across different starting points — with 18:2 imbalance, the result is stable because the optimum is always the same.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report

X = df[['amount', 'hour', 'is_foreign']].values
y = df['fraud'].values

model = LogisticRegression(random_state=42)
model.fit(X, y)
y_pred = model.predict(X)

print(confusion_matrix(y, y_pred))
print(classification_report(y, y_pred, target_names=['legit', 'fraud'], zero_division=0))

[[18  0]
 [ 2  0]]
              precision    recall  f1-score   support
        legit       0.90      1.00      0.95        18
        fraud       0.00      0.00      0.00         2
    accuracy                           0.90        20

The model converges to the same all-zero solution. Eighteen negatives dominate the gradient updates, so the loss-minimizing weights simply push every prediction toward the majority class. Both fraud samples land in the false-negative cell. The training accuracy is 0.90 — identical to a model that did nothing.

The mechanism: cross-entropy loss treats each sample as one term. With 18 negatives and 2 positives, the gradient signal from negatives is 9× the gradient signal from positives, and the classifier learns to ignore the minority class.

✓ Step 3 complete. A real logistic regression did the same thing as the always-0 baseline — the 18:2 ratio in the loss function drowned out the minority signal.

Root Causes and When It Occurs

Imbalance is a property of the data, but the severity and the cost of failure determine the right response.

Inherent rarity — fraud, rare disease diagnosis, equipment failure, ad click-through. The positive class is naturally uncommon because the event itself is rare.

Data collection bias — easier to label or observe the majority class. A medical dataset collected at a specialty clinic over-represents severe cases.

The question that changes the strategy: is the rare class just rare, or is detecting it critically important? In medical screening, missing 1 cancer case far outweighs falsely alarming 10 healthy patients. In spam filtering, the asymmetry reverses — a legitimate email in spam is worse than an extra spam in the inbox.

Different imbalance ratios demand different responses:

Evaluation Protocol

Resampling is for the training set, never the test set. Compute metrics on the original holdout.

n_splits=5 gives us 80/20 train/test splits while preserving the 18:2 ratio in each fold. shuffle=True randomizes the row order before splitting (otherwise sequential rows could create biased splits), and random_state=42 makes the shuffle reproducible.

from sklearn.model_selection import StratifiedKFold

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for fold, (train_idx, test_idx) in enumerate(skf.split(X, y), 1):
    X_train, X_test = X[train_idx], X[test_idx]
    y_train, y_test = y[train_idx], y[test_idx]
    print(f"Fold {fold}: train class counts = {np.bincount(y_train)}, test class counts = {np.bincount(y_test)}")

Fold 1: train class counts = [14  2], test class counts = [4 0]
Fold 2: train class counts = [15  1], test class counts = [3 1]
Fold 3: train class counts = [14  2], test class counts = [4 0]
Fold 4: train class counts = [14  2], test class counts = [4 0]
Fold 5: train class counts = [15  1], test class counts = [3 1]

Stratified k-fold preserves the 18:2 ratio in every fold. The n=20 anchor is too small for a meaningful cross-validation — fold 2 and fold 5 happen to have one fraud sample in the test set, the rest have zero. This is exactly the situation where resampling would inflate metrics: if you resample before splitting, the same fraud sample leaks into the training fold and gets evaluated on itself. Always split first.

Report all five metrics on the original holdout: precision, recall, F1, PR-AUC, and ROC-AUC. The fact that two of five folds have no positive samples in the test set is itself a finding — it tells you a 20-sample dataset cannot reliably evaluate a 10%-positive classifier.

✓ Step 4 complete. We set up stratified k-fold evaluation — resample only on training folds, never the test set — and learned that 20 samples is too few to reliably evaluate a 10%-positive classifier.

When It Works and When It Doesn't

The evaluation protocol here works well when class imbalance is between 2:1 and 100:1 and you have enough data for meaningful cross-validation. It breaks down when:

• You have fewer than ~50 positive samples — stratified folds will have empty positive bins, and precision-recall curves become unreliable
• The minority class is the natural majority in a sub-segment you care about — you may need to reframe as separate models per segment
• Cost asymmetry is extreme (e.g., medical diagnosis where a false negative costs 1000× a false positive) — metric thresholds need calibration, not just class weights

Metric Comparison

What Comes Next

This post diagnosed the problem: imbalanced data and the wrong metrics hide a model that learned nothing. The next post covers the fix — generating synthetic minority samples with SMOTE, the standard oversampling technique that creates new points along the line segments between existing minority samples instead of duplicating them.

Related Concepts

Backward: the baseline trap only makes sense once you understand classification metrics — precision, recall, F1 — and how a confusion matrix encodes them. A logistic regression on a balanced dataset is the natural predecessor.

Forward: the SMOTE family of techniques (SMOTE, Borderline-SMOTE, ADASYN, SMOTE-Tomek, SMOTE-ENN) all assume you have already decided that the minority class is the class of interest and that resampling will happen only on the training set. CTGAN and VAE-based synthesis take the same idea further by learning a generative model of the minority distribution.

Honest Limitations

• With fewer than ~50 positive samples, resampling cannot reliably help. The synthetic samples are interpolations of too few real points to add information. Use domain knowledge, anomaly detection, or active learning instead.
• Class-weight adjustments are not free. Setting class_weight='balanced' in a linear model can shift the decision boundary far from the data's actual support, especially when the minority class clusters tightly. Always check that the new boundary still makes physical sense.
• PR-AUC is unstable on tiny holdout sets. With only two fraud samples in the test fold, the precision-recall curve has at most three points. Report the curve, not just the area.

Test Your Understanding

1. Conceptual — On the 18:2 fraud anchor, why does the always-0 baseline achieve 0.90 accuracy but 0.0 recall? Which metric exposes the failure that accuracy hides?
2. Conceptual — What is the difference between a model that achieves 0.85 ROC-AUC and one that achieves 0.85 PR-AUC on a 99:1 dataset? Which is more informative about the model's positive-class performance?
3. Applied — Compute the balanced accuracy of a model that predicts 0 for the first 18 rows of the anchor and 1 for the last 2. How does it compare to the always-0 baseline?
4. Applied — On the anchor, why does StratifiedKFold(n_splits=5) produce test folds with either 0 or 1 fraud sample? What does this say about evaluating a 10%-positive classifier on 20 rows?
5. Edge case — A medical screening test has 99.5% specificity but only 60% sensitivity. The disease prevalence is 1 in 10,000. If a patient tests positive, what is the approximate probability they actually have the disease? Why does this matter for choosing the operating threshold?