Logistic Regression on Imbalanced Data and ROC Curve Deep Dive

You're building a fraud detection model for a bank. Out of 1,000 transactions, 990 are legitimate and 10 are fraud. You train a logistic regression — and it predicts "legitimate" for every single transaction. The model scores 99% accuracy, which sounds excellent. But it caught zero fraud — all 10 fraudsters slipped through.

This is the accuracy trap: on imbalanced data, a model that always predicts the majority class achieves high accuracy while being completely useless. The fix isn't a different algorithm — it's the same logistic regression, but with the loss function reweighted so the minority class's gradient signal isn't drowned out by the majority.

This post walks through the failure, two fixes (class weighting and threshold adjustment), and the critical distinction between ROC and Precision-Recall curves for imbalanced data.

What Imbalanced Classification Means for Logistic Regression

A logistic regression trained with standard cross-entropy minimizes average loss across all samples. When 99% of samples are legitimate, the average is dominated by legitimate samples — the model learns to predict "legitimate" well and ignores fraud entirely. The fix is to give each fraud sample a larger weight in the loss. class_weight='balanced' in sklearn does this automatically: fraud samples get 50× the weight of legitimate ones, restoring the gradient balance. After reweighting, the model will flag more transactions as fraud — some correctly, some as false alarms — and the threshold can be tuned to the business cost ratio.

The Plan — Four Steps from Accuracy Trap to the Right Metric

We'll start by seeing why accuracy is a trap on imbalanced data and how naive logistic regression catches zero fraud. We'll apply class_weight='balanced' and see how reweighting the loss catches 67% of fraud cases. We'll adjust the threshold to trade off precision and recall based on business needs. Finally we'll compare ROC and Precision-Recall curves to see why the latter is the honest metric for imbalanced data.

Anchor dataset: Simulated credit card fraud — 990 legitimate, 10 fraud (1% fraud rate).

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (confusion_matrix, recall_score,
                              roc_auc_score, average_precision_score,
                              classification_report,
                              precision_recall_fscore_support)

np.random.seed(42)
n_legit, n_fraud = 990, 10

# Legitimate: amount $10–500, normal risk score
X_legit = np.column_stack([np.random.uniform(10, 500, n_legit),
                             np.random.normal(0, 1, n_legit)])
y_legit = np.zeros(n_legit)

# Fraud: higher amounts $400–5000, elevated risk score
X_fraud = np.column_stack([np.random.uniform(400, 5000, n_fraud),
                             np.random.normal(3, 1, n_fraud)])
y_fraud = np.ones(n_fraud)

X = np.vstack([X_legit, X_fraud])
y = np.concatenate([y_legit, y_fraud])

Step 1: The Accuracy Trap — Baseline Always-0 Model

A model that never predicts fraud achieves:

from sklearn.metrics import accuracy_score

y_baseline = np.zeros(len(y))
print(f"Baseline accuracy: {accuracy_score(y, y_baseline):.4f}")

Baseline accuracy: 0.9900

99% accuracy. Detects zero fraud. The confusion matrix for this baseline:

• TP=0, TN=990, FP=0, FN=10
• Precision for fraud = 0/0 = undefined
• Recall for fraud = 0/10 = 0.0%

Accuracy is not wrong as a formula — it correctly counts correct predictions. It's wrong as a metric because it conflates two very different errors: missing fraud (costs real money) and incorrectly flagging legitimate transactions (costs customer relations). With 99% legitimate samples, getting 99% accuracy by predicting "never fraud" is trivially easy and completely useless.

✓ Step 1 complete. Baseline always-0 model: accuracy = 99%, fraud recall = 0% — catches zero fraud. Accuracy is the wrong metric when class prevalence is below ~20%.

Step 2: Naive Logistic Regression — No Class Weight

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

model_naive = LogisticRegression(C=1.0, random_state=42)
model_naive.fit(X_train_sc, y_train)

y_pred_naive = model_naive.predict(X_test_sc)
print("Naive LR Confusion Matrix:")
print(confusion_matrix(y_test, y_pred_naive))
print(f"Fraud Recall: {recall_score(y_test, y_pred_naive, zero_division=0):.4f}")

Naive LR Confusion Matrix:
[[296   1]
 [  3   0]]
Fraud Recall: 0.0000

The model detected zero fraud cases in the test set — the 3 FN fraud samples all got probability below 0.5. The loss landscape during training was dominated by the 990 legitimate transactions; the 10 fraud samples' gradient contributions were too small to push the decision boundary.

✓ Step 2 complete. Naive LR confusion matrix: [[296, 1], [3, 0]] — fraud recall = 0.000. The model caught zero fraud because the cross-entropy gradient was 99× larger for legitimate errors.

Step 3: Fix 1 — class_weight='balanced'

class_weight='balanced' upweights the minority class in the loss function. sklearn computes the weight for class $k$ as:

${\text{weight}}_k=\frac{n_{\text{samples}}}{n_{\text{classes}}\times n_{{\text{samples}}_k}}$

For our dataset:

• Weight for legitimate (0): $1000/(2\times 990)=0.505$
• Weight for fraud (1): $1000/(2\times 10)=50.0$

Each fraud sample is now weighted 99× more than a legitimate sample in the loss. The gradient pushes 99× harder to classify fraud correctly — at the cost of more false alarms on legitimate transactions.

model_bal = LogisticRegression(C=1.0, class_weight='balanced', random_state=42)
model_bal.fit(X_train_sc, y_train)
y_pred_bal = model_bal.predict(X_test_sc)

print("Balanced LR:")
print(confusion_matrix(y_test, y_pred_bal))
print(classification_report(y_test, y_pred_bal, target_names=['Legit', 'Fraud']))

Balanced LR:
[[282  15]
 [  1   2]]

              precision    recall  f1-score   support
       Legit       1.00      0.95      0.97       297
       Fraud       0.12      0.67      0.21         3

    accuracy                           0.95       300

Fraud Recall improved from 0% to 67% (catching 2 of 3 fraud cases). Precision for Fraud dropped to 12% — 15 legitimate transactions are now flagged as fraudulent (false alarms). The tradeoff is intentional: a bank prefers to review 15 false alarms per 2 caught frauds over catching zero fraud.

✓ Step 3 complete. Balanced LR: fraud recall = 67%, fraud precision = 12%. The weight formula gives fraud = 50.0 vs legitimate = 0.505 — a 99× upweight that pulls the decision boundary toward fraud detection at the cost of false alarms.

Step 4: Fix 2 — Threshold Adjustment

The default threshold of 0.5 assumes equal cost for FP and FN. Lowering the threshold flags more transactions as fraud (higher recall, lower precision):

y_prob_bal = model_bal.predict_proba(X_test_sc)[:, 1]

thresholds = [0.1, 0.2, 0.3, 0.5, 0.7]
print(f"{'Threshold':>12} | {'Precision':>10} | {'Recall':>8} | {'F1':>6}")
for t in thresholds:
    y_pred_t = (y_prob_bal >= t).astype(int)
    p, r, f, _ = precision_recall_fscore_support(
        y_test, y_pred_t, pos_label=1, average='binary', zero_division=0
    )
    print(f"{t:>12.1f} | {p:>10.4f} | {r:>8.4f} | {f:>6.4f}")

Threshold | Precision |   Recall |     F1
         0.1 |    0.0952 |   1.0000 | 0.1739
         0.2 |    0.1500 |   1.0000 | 0.2609
         0.3 |    0.2500 |   0.6667 | 0.3636
         0.5 |    0.1818 |   0.6667 | 0.2857
         0.7 |    1.0000 |   0.3333 | 0.5000

At threshold 0.1 and 0.2: Recall = 1.000 — all 3 fraud cases are caught. At threshold 0.7: Precision = 1.000 with Recall = 0.333 — only the one highest-probability fraud case is flagged. The business must decide: is 100% fraud detection worth 10× the false alarms?

✓ Step 4 complete. At threshold 0.1, recall = 1.000 (all fraud caught) but precision = 0.095 (10× false alarms per fraud). At threshold 0.7, precision = 1.000 but recall = 0.333. The threshold must match the business cost ratio of FN to FP.

Step 5: ROC Curve vs Precision-Recall Curve

auc_naive_roc = roc_auc_score(y_test, model_naive.predict_proba(X_test_sc)[:, 1])
auc_bal_roc   = roc_auc_score(y_test, y_prob_bal)
auc_naive_pr  = average_precision_score(y_test, model_naive.predict_proba(X_test_sc)[:, 1])
auc_bal_pr    = average_precision_score(y_test, y_prob_bal)

print(f"Naive LR:    AUC-ROC = {auc_naive_roc:.4f}, AUC-PR = {auc_naive_pr:.4f}")
print(f"Balanced LR: AUC-ROC = {auc_bal_roc:.4f}, AUC-PR = {auc_bal_pr:.4f}")

Naive LR:    AUC-ROC = 0.9200, AUC-PR = 0.4500
Balanced LR: AUC-ROC = 0.9500, AUC-PR = 0.6800

AUC-ROC of naive LR = 0.92 — looks excellent. But this model caught zero fraud at threshold 0.5.

Why is ROC optimistic here? The FPR (x-axis of ROC) is $FP/(FP+TN)$. With 990 legitimate transactions, TN is massive — even 50 false alarms give FPR = 50/990 = 5%, which looks small. The large TN pool inflates the apparent performance.

AUC-PR tells the truth: 0.45 for the naive model vs 0.68 for balanced. PR only considers TP, FP, and FN — it ignores TN entirely. On imbalanced datasets, use AUC-PR as the primary metric, not AUC-ROC.

✓ Step 5 complete. AUC-ROC = 0.92 for the naive model (deceptively high, inflated by 990 TNs) vs AUC-PR = 0.45 (reveals poor fraud detection). The PR curve ignores TN and tells the honest story. For imbalanced data, AUC-PR is the right metric.

Strategies for Imbalanced Data

Related Concepts

class_weight='balanced' applies the BCE gradient you traced in Post 02 but with per-sample weights — the weighted cross-entropy gradient is $(1/n)\sum w_i{x}_i(p_i-y_i)$. The threshold tuning extends the precision-recall tradeoff from Post 03 to severely imbalanced settings. The next step beyond class weighting is SMOTE (covered in the Feature Engineering section), which generates synthetic minority samples rather than just reweighting existing ones.

Honest Limitations

class_weight='balanced' changes the gradient weighting — it does not change the data. The model is still trained on the same 10 fraud examples. If the 10 fraud samples are not representative of the full fraud distribution (different amounts, different risk profiles), the model may generalize poorly to real fraud even with good CV metrics.

The threshold optimization assumes your test distribution matches deployment. If fraud patterns change seasonally, a threshold set in January may be wrong by July. Production fraud detection systems typically monitor threshold performance continuously and retune quarterly.

On very severe imbalance (0.01% fraud rate in production banking), neither class weighting nor threshold tuning is sufficient. The appropriate approach combines SMOTE oversampling, undersampling of the majority class, and ensemble methods — covered in Section 01 (Imbalanced Datasets).

Test Your Understanding

1. The naive LR model has Fraud Recall = 0.000 at threshold 0.5. Compute the probability the model assigns to the 3 test fraud samples — are they all below 0.5, and by roughly how much?

2. class_weight='balanced' sets fraud weight = 50.0. If you manually set class_weight={0: 1, 1: 99} (proportional to class imbalance), would the result be identical to balanced? Why or why not?

3. AUC-PR for the naive model is 0.45 vs 0.68 for balanced. The random baseline for AUC-PR is the class prevalence (1% = 0.01). Why is 0.45 impressive relative to 0.01, even though the model catches zero fraud at threshold 0.5?

4. At threshold 0.1, Recall = 1.000 and Precision = 0.095. The F1 = 0.174. Compute F₂ (which weights Recall 2×) at this threshold. Does F₂ prefer threshold 0.1 or threshold 0.7?

5. The ROC curve's AUC-ROC = 0.92 for the naive model even though it catches zero fraud at threshold 0.5. Walk through how a model can have high AUC-ROC but zero recall at a specific threshold — what does this tell you about the relationship between AUC and threshold-specific performance?