Logistic Regression on Imbalanced Data and ROC Curve Deep Dive

A logistic regression model trained on 99% legitimate transactions will learn one thing: everything is legitimate. Not because it's broken — because the loss is dominated by the majority class. The gradient of the cross-entropy loss has 990 small pushes from legitimate samples for every 10 large pushes from fraud. Fix this by reweighting the loss, not by changing the algorithm.

Anchor dataset: Simulated credit card fraud — 990 legitimate, 10 fraud (1% fraud rate).

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (confusion_matrix, recall_score,
                              roc_auc_score, average_precision_score,
                              classification_report,
                              precision_recall_fscore_support)

np.random.seed(42)
n_legit, n_fraud = 990, 10

# Legitimate: amount $10–500, normal risk score
X_legit = np.column_stack([np.random.uniform(10, 500, n_legit),
                             np.random.normal(0, 1, n_legit)])
y_legit = np.zeros(n_legit)

# Fraud: higher amounts $400–5000, elevated risk score
X_fraud = np.column_stack([np.random.uniform(400, 5000, n_fraud),
                             np.random.normal(3, 1, n_fraud)])
y_fraud = np.ones(n_fraud)

X = np.vstack([X_legit, X_fraud])
y = np.concatenate([y_legit, y_fraud])

The Accuracy Trap — The Baseline Always-0 Model

A model that never predicts fraud achieves:

from sklearn.metrics import accuracy_score

y_baseline = np.zeros(len(y))
print(f"Baseline accuracy: {accuracy_score(y, y_baseline):.4f}")

99% accuracy. Detects zero fraud. The confusion matrix for this baseline:

• TP=0, TN=990, FP=0, FN=10
• Precision for fraud = 0/0 = undefined
• Recall for fraud = 0/10 = 0.0%

Accuracy is not wrong as a formula — it correctly counts correct predictions. It's wrong as a metric because it conflates two very different errors: missing fraud (costs real money) and incorrectly flagging legitimate transactions (costs customer relations). With 99% legitimate samples, getting 99% accuracy by predicting "never fraud" is trivially easy and completely useless.

Naive Logistic Regression (No Class Weight)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc  = scaler.transform(X_test)

model_naive = LogisticRegression(C=1.0, random_state=42)
model_naive.fit(X_train_sc, y_train)

y_pred_naive = model_naive.predict(X_test_sc)
print("Naive LR Confusion Matrix:")
print(confusion_matrix(y_test, y_pred_naive))
print(f"Fraud Recall: {recall_score(y_test, y_pred_naive, zero_division=0):.4f}")

The model detected zero fraud cases in the test set — the 3 FN fraud samples all got probability below 0.5. The loss landscape during training was dominated by the 990 legitimate transactions; the 10 fraud samples' gradient contributions were too small to push the decision boundary.

Fix 1: class_weight='balanced'

class_weight='balanced' upweights the minority class in the loss function. sklearn computes the weight for class $k$ as:

${\text{weight}}_k=\frac{n_{\text{samples}}}{n_{\text{classes}}\times n_{{\text{samples}}_k}}$

For our dataset:

• Weight for legitimate (0): $1000/(2\times 990)=0.505$
• Weight for fraud (1): $1000/(2\times 10)=50.0$

Each fraud sample is now weighted 99× more than a legitimate sample in the loss. The gradient pushes 99× harder to classify fraud correctly — at the cost of more false alarms on legitimate transactions.

model_bal = LogisticRegression(C=1.0, class_weight='balanced', random_state=42)
model_bal.fit(X_train_sc, y_train)
y_pred_bal = model_bal.predict(X_test_sc)

print("Balanced LR:")
print(confusion_matrix(y_test, y_pred_bal))
print(classification_report(y_test, y_pred_bal, target_names=['Legit', 'Fraud']))

Fraud Recall improved from 0% to 67% (catching 2 of 3 fraud cases). Precision for Fraud dropped to 12% — 15 legitimate transactions are now flagged as fraudulent (false alarms). The tradeoff is intentional: a bank prefers to review 15 false alarms per 2 caught frauds over catching zero fraud.

Fix 2: Threshold Adjustment

The default threshold of 0.5 assumes equal cost for FP and FN. Lowering the threshold flags more transactions as fraud (higher recall, lower precision):

y_prob_bal = model_bal.predict_proba(X_test_sc)[:, 1]

thresholds = [0.1, 0.2, 0.3, 0.5, 0.7]
print(f"{'Threshold':>12} | {'Precision':>10} | {'Recall':>8} | {'F1':>6}")
for t in thresholds:
    y_pred_t = (y_prob_bal >= t).astype(int)
    p, r, f, _ = precision_recall_fscore_support(
        y_test, y_pred_t, pos_label=1, average='binary', zero_division=0
    )
    print(f"{t:>12.1f} | {p:>10.4f} | {r:>8.4f} | {f:>6.4f}")

At threshold 0.1 and 0.2: Recall = 1.000 — all 3 fraud cases are caught. At threshold 0.7: Precision = 1.000 with Recall = 0.333 — only the one highest-probability fraud case is flagged. The business must decide: is 100% fraud detection worth 10× the false alarms?

ROC Curve vs Precision-Recall Curve for Imbalanced Data

auc_naive_roc = roc_auc_score(y_test, model_naive.predict_proba(X_test_sc)[:, 1])
auc_bal_roc   = roc_auc_score(y_test, y_prob_bal)
auc_naive_pr  = average_precision_score(y_test, model_naive.predict_proba(X_test_sc)[:, 1])
auc_bal_pr    = average_precision_score(y_test, y_prob_bal)

print(f"Naive LR:    AUC-ROC = {auc_naive_roc:.4f}, AUC-PR = {auc_naive_pr:.4f}")
print(f"Balanced LR: AUC-ROC = {auc_bal_roc:.4f}, AUC-PR = {auc_bal_pr:.4f}")

AUC-ROC of naive LR = 0.92 — looks excellent. But this model caught zero fraud at threshold 0.5.

Why is ROC optimistic here? The FPR (x-axis of ROC) is $FP/(FP+TN)$. With 990 legitimate transactions, TN is massive — even 50 false alarms give FPR = 50/990 = 5%, which looks small. The large TN pool inflates the apparent performance.

AUC-PR tells the truth: 0.45 for the naive model vs 0.68 for balanced. PR only considers TP, FP, and FN — it ignores TN entirely. On imbalanced datasets, use AUC-PR as the primary metric, not AUC-ROC.

Strategies for Imbalanced Data

Related Concepts and Honest Limitations

class_weight='balanced' changes the gradient weighting — it does not change the data. The model is still trained on the same 10 fraud examples. If the 10 fraud samples are not representative of the full fraud distribution (different amounts, different risk profiles), the model may generalize poorly to real fraud even with good CV metrics.

The threshold optimization assumes your test distribution matches deployment. If fraud patterns change seasonally, a threshold set in January may be wrong by July. Production fraud detection systems typically monitor threshold performance continuously and retune quarterly.

On very severe imbalance (0.01% fraud rate in production banking), neither class weighting nor threshold tuning is sufficient. The appropriate approach combines SMOTE oversampling, undersampling of the majority class, and ensemble methods — covered in Section 01 (Imbalanced Datasets).

Test Your Understanding

1. The naive LR model has Fraud Recall = 0.000 at threshold 0.5. Compute the probability the model assigns to the 3 test fraud samples — are they all below 0.5, and by roughly how much?

2. class_weight='balanced' sets fraud weight = 50.0. If you manually set class_weight={0: 1, 1: 99} (proportional to class imbalance), would the result be identical to balanced? Why or why not?

3. AUC-PR for the naive model is 0.45 vs 0.68 for balanced. The random baseline for AUC-PR is the class prevalence (1% = 0.01). Why is 0.45 impressive relative to 0.01, even though the model catches zero fraud at threshold 0.5?

4. At threshold 0.1, Recall = 1.000 and Precision = 0.095. The F1 = 0.174. Compute F₂ (which weights Recall 2×) at this threshold. Does F₂ prefer threshold 0.1 or threshold 0.7?

5. The ROC curve's AUC-ROC = 0.92 for the naive model even though it catches zero fraud at threshold 0.5. Walk through how a model can have high AUC-ROC but zero recall at a specific threshold — what does this tell you about the relationship between AUC and threshold-specific performance?