Imbalanced Datasets

To ground every technique in the same numbers, use a fraud-detection anchor: 20 transactions with only 4 fraudulent (minority) and 16 legitimate (majority). The features — amount and hour — are simple enough that every resampling step can be traced by hand.

import numpy as np
# 20 samples: 4 fraud (minority), 16 legitimate (majority)
# features: [amount, hour]
X = np.array([
    [120, 280], [160, 230], [190, 300], [140, 190],  # fraud
    [90, 150], [80, 140], [85, 100], [75, 180],       # legit
    [100, 120], [70, 160], [60, 200], [95, 80],       # legit
    [110, 140], [65, 170], [105, 90], [55, 210],      # legit
    [115, 130], [78, 145], [88, 110], [72, 155],      # legit
])
y = np.array([1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
print(f"Class distribution: {y.sum()} fraud, {len(y)-y.sum()} legitimate")

Class distribution: 4 fraud, 16 legitimate

With an 80:20 ratio, accuracy is already a misleading 80% for a "predict all legitimate" baseline. Every technique in this section will be compared against this baseline.

The three pillars of imbalance handling

Data-level — fix the imbalance in the data itself before training. Algorithm-level — make the model aware of the imbalance during training. Evaluation — measure the right thing, not accuracy.

Oversampling techniques (tabular)

Random oversampling — duplicate minority samples randomly. Simple, but causes overfitting on the same points.

SMOTE (Synthetic Minority Oversampling Technique) — generates synthetic samples by interpolating between a minority point and its k nearest neighbors. The workhorse of oversampling. A key distinction: it creates samples in feature space, not pixel/text space, so it's purely for tabular/numerical data.

ADASYN (Adaptive Synthetic Sampling) — like SMOTE, but generates more synthetic samples in harder-to-learn regions (near the decision boundary). More adaptive than vanilla SMOTE.

Borderline-SMOTE — only oversamples minority points that are near the decision boundary (borderline examples), ignoring easy ones deep inside the minority cluster.

SVMSMOTE — uses SVM support vectors to guide where synthetic samples are placed.

Undersampling techniques

Random undersampling — drop majority class samples randomly. Risk: loses potentially useful information.

Tomek links — removes the majority sample from a pair of very close minority-majority samples. Cleans the boundary rather than heavily reducing data.

NearMiss — selects majority samples whose average distance to the nearest minority samples is smallest (version-1, 2, or 3). More principled than random.

Cluster centroids — replace a cluster of majority samples with their centroid. Reduces majority while preserving structure.

Edited Nearest Neighbors (ENN) — removes majority samples misclassified by their k-nearest neighbors. Good for noise removal.

Algorithm-level

class_weight parameter — built into sklearn's Logistic Regression, SVM, RandomForest, etc. Setting class_weight='balanced' automatically inversely weights each class by frequency. One of the easiest first things to try.

Focal Loss — introduced by Facebook AI for object detection (RetinaNet). Down-weights the loss on easy (well-classified) samples and focuses training on hard ones. Key formula: FL(p_t) = -(1-p_t)^γ * log(p_t). γ controls the focus. Standard in computer vision imbalance.

XGBoost scale_pos_weight — set to negative_count / positive_count. Tells the boosting algorithm to penalize missing minority class more.

Threshold tuning — after training, shift the classification threshold from default 0.5 toward 0.3 or lower to favor recall of the minority class.

Ensemble methods for imbalance

BalancedBaggingClassifier — bagging where each bootstrap sample is balanced before training each base estimator.

EasyEnsemble — creates multiple balanced datasets by random undersampling and trains a model on each; combines by averaging.

RUSBoost — combines Random Undersampling with AdaBoost. At each boosting round, the majority class is undersampled.

BalancedRandomForest — like RandomForest but each tree is trained on a balanced bootstrap sample.

By data type

Tabular

SMOTE variants are the go-to. Also: CTGAN (Conditional GAN for tabular data) to generate realistic minority samples. Feature engineering can sometimes help expose a clearer signal for the minority class.

Text / NLP

• EDA (Easy Data Augmentation) — synonym replacement, random insertion, random swap, random deletion on minority class sentences.
• Back-translation — translate text to French then back to English; paraphrase at low cost.
• LLM paraphrasing — use GPT/Claude to generate paraphrases of minority class examples. Increasingly common and very effective.
• Few-shot fine-tuning — fine-tune a pretrained model (BERT etc.) which already has rich representations; needs far less minority data.
• Oversampling at the embedding level — SMOTE in embedding space rather than raw text.

Computer vision

• Geometric augmentation — flips, crops, rotations, color jitter on minority class images. Cheapest and most effective first step.
• Mixup / CutMix — blend two images and their labels; forces the model to learn smoother decision boundaries.
• GAN / Diffusion synthesis — generate realistic minority class images using a GAN (DCGAN, StyleGAN) or diffusion model. Expensive but powerful for rare-class detection.
• Transfer learning — pretrain on ImageNet, fine-tune on your imbalanced dataset. The pretrained backbone needs far fewer minority samples to generalize.
• Focal loss — especially important for object detection where background heavily outnumbers objects.

Time series

• SMOTE-TS — SMOTE adapted for temporal structure, interpolates between temporal windows.
• Window sliding — create more training windows from minority-class events (shorter stride).
• Anomaly detection framing — reframe as one-class classification (learn what "normal" looks like, flag everything else). Useful when minority is extremely rare (fraud, fault detection).
• Time-aware augmentation — warp, scale, or add noise to temporal minority sequences.

Evaluation metrics for imbalanced data

Never use accuracy with imbalanced data. If 99% of data is class 0, predicting class 0 always gives 99% accuracy while being useless.

Precision = of all predicted positives, how many are truly positive. Recall = of all actual positives, how many did we catch. (Often more important in fraud/medical.) F1 = harmonic mean of precision and recall. PR-AUC (Area Under Precision-Recall Curve) — better than ROC-AUC for highly imbalanced data because ROC-AUC can be misleadingly high. MCC (Matthews Correlation Coefficient) — robust single metric even with extreme imbalance. G-mean = √(Sensitivity × Specificity). Stratified k-fold — essential; ensures each fold maintains the class distribution ratio.

Related Concepts

Backward: This post assumes you understand standard classification metrics (accuracy, precision, recall, F1), how logistic regression and tree-based models make predictions, and basic feature engineering (handling missing values, outliers, encoding).

Forward: Each technique listed here gets its own deep-dive: SMOTE and its variants (Borderline-SMOTE, ADASYN, SMOTE-Tomek, SMOTE-ENN), cost-sensitive learning (class weights, focal loss), and ensemble approaches (BalancedBagging, RUSBoost). Evaluation metrics for imbalance — PR-AUC, MCC — are covered in detail in the classification metrics post.

Honest Limitations

1. No silver bullet. Every technique here trades off something. Oversampling can introduce noisy synthetic points; undersampling discards potentially useful majority data; cost-sensitive methods assume you know the true cost ratio.
2. Effectiveness depends on separability. When classes overlap heavily, no resampling technique cleanly separates them — the model still sees ambiguous boundary points regardless of class ratios.
3. Synthetic data has limits. SMOTE and its variants work in feature space — they cannot generate realistic text passages, images, or time-series segments. For those domains, domain-specific augmentation (EDA, geometric transforms, window sliding) is required.

Test Your Understanding

1. A credit-card fraud dataset has 100,000 legitimate transactions and 500 fraudulent ones. Accuracy on a model that predicts "legitimate" every time is 99.5%. Why is this number useless, and which metric should replace it?

2. You apply SMOTE to a dataset where three minority samples form a straight-line cluster in feature space. What shape will the synthetic minority points roughly follow? What happens if two of those minority samples are actually outliers?

3. A colleague says "I always use Random Undersampling first because it keeps the dataset small." Under what specific conditions does this advice backfire — i.e., when does removing majority samples actually hurt performance?

4. For a text classification task with 1,000 positive reviews and 10 negative reviews, can you apply SMOTE directly on the raw text? If not, at what stage of the pipeline should you oversample, and how would CTGAN differ from SMOTE in this context?

5. Focal loss introduces a focusing parameter γ. What happens to the loss function as γ → ∞? What about γ = 0? Sketch the behavior — a verbal description is fine.