Handling Outliers

You're analyzing your company's e-commerce orders. Nine of the past ten customers spent between $78and$110 — a tight, comfortable cluster. Customer 7 spent $4,800. Your first instinct is to delete it — clearly a data entry error, right? But what if it's a real B2B purchase? Remove it, and you teach your model that high-value customers don't exist. Keep it, and one point can distort your entire regression. This is the outlier dilemma — and answering it wrong costs your business real money.

The challenge is that outliers aren't a single problem with a single fix. An error outlier — a sensor glitch, a copy-paste blunder — should be removed or corrected. A valid extreme observation — a genuine black-swan event or a high-value customer segment you need to forecast accurately — demands a gentler touch. Treat them the same way and you'll either keep garbage in your model or throw away your most interesting signal.

This post walks through three detection methods and four treatment strategies so you can diagnose the outlier type and respond appropriately. Each method is demonstrated on the same small dataset where you can verify every fence, z-score, and threshold by hand.

The Anchor

import pandas as pd
import numpy as np

data = {
    'customer_id': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    'order_value': [85, 92, 78, 110, 95, 88, 4800, 102, 79, 96],
    'items_count': [3, 4, 2, 5, 3, 4, 2, 4, 3, 4],
    'session_min': [12, 18, 8, 22, 15, 14, 6, 19, 11, 16],
}
df = pd.DataFrame(data)
# Customer 7: order_value=4800, clearly anomalous vs the 78-110 range

Every detection calculation, SVG, and code block uses these exact 10 rows.

The Plan — Three Detection Methods and Four Treatments

We'll walk through three detection methods (IQR, z-score, Isolation Forest) on the same 10-row e-commerce anchor, then evaluate four treatment strategies (remove, cap, log transform, keep) depending on what caused the outlier.

What Is an Outlier

A statistical outlier is a data point that lies far from the bulk of the distribution. There are two types that require completely different responses:

• Error outlier — data entry mistake, sensor glitch, copy-paste error. Should be removed or corrected.
• Valid extreme observation — a real high-value customer, a genuine black-swan event. Removing it loses signal and biases the model against the segment that produced it.

This is NOT the same as a missing value — an outlier is present but extreme, not absent. The right treatment depends entirely on which type you have.

Why an Outlier Breaks Standard Statistics

Why outliers harm linear regression: OLS minimizes the sum of squared errors, and a single large residual gets squared into a much larger penalty. The fitted line bends toward the outlier to reduce its squared cost, distorting the slope for all other points.

Fit a simple line predicting order_value from session_min and watch the slope change when customer 7 is included:

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

X = df[['session_min']].values
y = df['order_value'].values

# With all 10 customers
m_all = LinearRegression().fit(X, y)
pred_all = m_all.predict(X)
mse_all = mean_squared_error(y, pred_all)
print(f"With all 10: slope={m_all.coef_[0]:.2f}, MSE={mse_all:.1f}")

# Without customer 7
mask = df['customer_id'] != 7
m_clean = LinearRegression().fit(X[mask], y[mask])
mse_clean = mean_squared_error(y[mask], m_clean.predict(X[mask]))
print(f"Without customer 7: slope={m_clean.coef_[0]:.2f}, MSE={mse_clean:.1f}")

With all 10: slope=85.81, MSE=2053032.6
Without customer 7: slope=0.42, MSE=104.8

The slope swings from 0.42 (without the outlier) to 85.81 (with it) — a 200× change driven entirely by one point. The MSE goes from 105 to over 2 million. The model with the outlier is essentially useless for prediction on any of the other 9 customers.

Visually, the two regression lines diverge from the same y-intercept and fan out at customer 7's extreme point:

The red dashed line is forced to reach the lone point at (6, 4800). The green line passes through the cluster. The difference in slope is the entire story.

✓ Step 1 complete. The outlier (customer 7, $4800) changed the regression slope from 0.42 to 85.81 — a 200× swing driven by one point. Without the outlier, MSE was 105; with it, MSE exceeded 2 million.

Detecting with IQR (Interquartile Range)

The IQR method flags points that sit unusually far from the middle 50% of the data. Compute the 25th and 75th percentiles, then set fences at Q1 − 1.5×IQR and Q3 + 1.5×IQR.

Sort order_value: [78, 79, 85, 88, 92, 95, 96, 102, 110, 4800]

Let's compute the fences step by step. First we sort the values, then find the quartiles:

Customer 7's value of 4800 is 4677.4 units above the upper fence — clearly flagged. No other customer is flagged (next highest is 110, well under 122.625).

Q1, Q3 = 85.75, 100.5
IQR = Q3 - Q1
lower, upper = Q1 - 1.5*IQR, Q3 + 1.5*IQR
outliers = df[(df['order_value'] < lower) | (df['order_value'] > upper)]
print(f"Lower fence: {lower}, Upper fence: {upper}")
print(f"Outliers flagged:\n{outliers[['customer_id', 'order_value']]}")

Lower fence: 63.625, Upper fence: 122.625
Outliers flagged:
   customer_id  order_value
6            7         4800

The box-and-whisker representation shows the same picture: the box is the IQR, the whiskers extend to the fence, and customer 7 sits far past the upper whisker.

The box (Q1 to Q3) covers 85.75 to 100.5. The whiskers extend to the fences. Customer 7's dot at 4800 is drawn at the fence position with a label noting the true value is far beyond the chart.

✓ Step 2a complete. The IQR method flagged customer 7 ($4800>$122.63 upper fence). The quartile-based approach is robust — the Q1 and Q3 values are unaffected by the extreme point.

Detecting with Z-Score

Z-score measures how many standard deviations a point sits from the mean: z = (x − μ) / σ. The common threshold is |z| > 3.

For the anchor:

The z-score starts with the mean and standard deviation. Here's the issue — with the outlier included, both statistics are pulled toward the extreme value:

The mean is 562.5, far above the cluster center of ~90. The outlier inflated the mean by 6×. The standard deviation is also inflated, which makes it harder to detect the outlier — this is the masking effect.

Z-scores for every customer:

mean = df['order_value'].mean()
std = df['order_value'].std()
df['z_score'] = (df['order_value'] - mean) / std
df[['customer_id', 'order_value', 'z_score']]

customer_id  order_value   z_score
0            1           85    -0.338
1            2           92    -0.333
2            3           78    -0.343
3            4          110    -0.320
4            5           95    -0.333
5            6           88    -0.338
6            7         4800     2.999
7            8          102    -0.326
8            9           79    -0.343
9           10           96    -0.330

Customer 7's z = 2.999 — just under the strict |z| > 3 threshold. In practice, |z| > 2.5 is commonly used to catch the same point, and 3.0 is conservative. The "just under 3" case is exactly the masking effect at work: had the mean and std been computed without customer 7, the z for 4800 against a clean mean of ~91 and clean std of ~10 would be ≈ 470 — vastly over threshold.

The masking effect matters because the very statistic used to detect the outlier is contaminated by it. A robust z-score (using median and MAD instead of mean and std) avoids this.

✓ Step 2b complete. The z-score method gave customer 7 a score of 3.00 — right at the |z|>3 threshold. The masking effect: the outlier inflates the mean (562.5 vs true center ~90) and std, making itself harder to detect.

Detecting with Isolation Forest

Isolation Forest takes a model-based approach: outliers are easier to isolate, so they require fewer random splits in a random tree to separate from the rest. Build many random trees; samples isolated in shallower paths get higher anomaly scores.

contamination=0.1 sets the expected proportion of outliers — 10% of the dataset, which is 1 out of 10 rows, matching our expectation that customer 7 is the only anomaly. random_state=42 makes the random tree splits reproducible.

from sklearn.ensemble import IsolationForest

iso = IsolationForest(contamination=0.1, random_state=42)
features = df[['order_value', 'items_count', 'session_min']].values
iso.fit(features)
df['anomaly'] = iso.predict(features)
df[['customer_id', 'order_value', 'anomaly']]

customer_id  order_value  anomaly
0            1           85        1
1            2           92        1
2            3           78        1
3            4          110        1
4            5           95        1
5            6           88        1
6            7         4800       -1
7            8          102        1
8            9           79        1
9           10           96        1

Output -1 = anomaly, 1 = normal. Customer 7 is the only point flagged. With contamination=0.1 (the algorithm's prior on the fraction of outliers), the model expects about 10% of points to be anomalies — 1 out of 10 matches the prior.

Use Isolation Forest when:

• Data is high-dimensional (IQR and z-score operate on one feature at a time)
• Outliers form clusters of their own (a sub-population, not just extreme points on a single variable)
• The relationship between features carries outlier signal (e.g., 4800 with 2 items is a different anomaly than 4800 with 50 items)

✓ Step 2c complete. Isolation Forest flagged customer 7 as the only anomaly. Unlike the univariate methods, it can detect multivariate outliers — a $4800orderwith2itemsismoresuspiciousthan$4800 with 50 items.

Comparison of Detection Methods

✓ Step 2 complete. We have three complementary detection methods — IQR (robust, univariate), z-score (parametric, masking-prone), and Isolation Forest (multivariate, black-box).

Remove the outlier

The most direct response when the outlier is a confirmed error — wrong unit, copy-paste mistake, sensor glitch.

df_clean = df[df['order_value'] < 120]
print(f"Rows remaining: {df_clean.shape[0]}")

Rows remaining: 9

Nine rows remain. The model now trains on a clean distribution with no extreme values.

Warning: removing a valid extreme observation introduces survivorship bias. If customer 7 is a high-value B2B client and the model never sees them, the model will systematically under-predict revenue for that segment. The same logic applies to fraud, churn, and equipment failure — exactly the cases where the rare class is the class of interest.

✓ Step 3a complete. Removing customer 7 left 9 rows. The model now trains on a clean distribution, but if the outlier is a genuine high-value customer, we've introduced survivorship bias.

Winsorize: cap extreme values

Winsorizing replaces values beyond the [p5, p95] range with the percentile values themselves. The shape of the distribution is preserved, but extreme values are pulled back to a sane range.

For the anchor, the 95th percentile of order_value is around 579 (numpy interpolation). Customer 7's 4800 is replaced with 579.

from scipy.stats.mstats import winsorize

df['order_value_winsorized'] = winsorize(df['order_value'], limits=[0.05, 0.05])
print(df[['customer_id', 'order_value', 'order_value_winsorized']].iloc[6])

customer_id                   7
order_value                4800
order_value_winsorized    579.0

The other 9 customers are unchanged. Customer 7's 4800 is now 579.0 — the value is preserved as "extreme" but no longer dominates the scale.

Winsorizing keeps the row in the dataset (so the segment is represented) and prevents one point from distorting the model. The cost: the model sees a slightly less extreme value than the real data, which under-weights the tail.

✓ Step 3b complete. Winsorizing capped $4800at$579 — preserving the row as extreme but preventing one point from dominating the model's scale.

Log transform: compress the tail

A log transform compresses the right tail multiplicatively. The gap between 4800 and 85 in linear units (4715) becomes a gap of 4.02 in log units (log(4801) − log(86) = 8.477 − 4.454 = 4.022).

df['order_value_log'] = np.log1p(df['order_value'])
print(df[['customer_id', 'order_value', 'order_value_log']].to_string(index=False))
print(f"\nLog gap (4 vs 7): {df['order_value_log'].iloc[6] - df['order_value_log'].iloc[0]:.3f}")
print(f"Linear gap: {df['order_value'].iloc[6] - df['order_value'].iloc[0]}")

customer_id  order_value  order_value_log
          1           85         4.454
          2           92         4.522
          3           78         4.357
          4          110         4.700
          5           95         4.554
          6           88         4.477
          7         4800         8.477
          8          102         4.625
          9           79         4.431
         10           96         4.564

Log gap (4 vs 7): 4.022
Linear gap: 4715

The 4715-unit linear gap is now 4.0 log units. Customer 7 is still visibly the largest, but no longer 50× larger than everyone else. The model can fit a relationship across the full range without one point dominating the loss.

Use log transforms on right-skewed data: income, revenue, count data, anything where the scale is exponential. The transform assumes all values are positive — apply log1p to handle zeros.

✓ Step 3c complete. The log transform compressed the gap from 4715 linear units to 4.02 log units — customer 7 is still visibly the largest but no longer 50× larger than everyone else.

Keep: when the outlier carries signal

Sometimes the right response is no transformation at all. If customer 7 is a real B2B order, removing or capping them teaches the model that this segment does not exist. The high-value segment will be systematically under-predicted, and revenue forecasts will miss it.

Two ways to keep the outlier and still get a usable model:

• Robust regression — use Huber loss or MAE instead of MSE. These losses penalize large errors less aggressively, so the outlier does not dominate the fit.

To see how a robust estimator handles the same data, we compare Huber regression against ordinary least squares. Huber loss combines squared error (like OLS) for small residuals and absolute error (like MAE) for large residuals — the transition point is controlled by epsilon (default=1.35).

from sklearn.linear_model import HuberRegressor, LinearRegression
from sklearn.metrics import mean_absolute_error

huber = HuberRegressor().fit(X, y)
mae_huber = mean_absolute_error(y, huber.predict(X))
mse_huber = mean_squared_error(y, huber.predict(X))
print(f"Huber MAE: {mae_huber:.1f}, MSE: {mse_huber:.1f}")

lin = LinearRegression().fit(X, y)
mse_lin = mean_squared_error(y, lin.predict(X))
print(f"Linear MSE on same data: {mse_lin:.1f}")

Huber MAE: 504.0, MSE: 2052446.8
Linear MSE on same data: 2053032.6

Huber barely changes the MSE here because the outlier is so extreme that even Huber caps its influence. With more moderate outliers, the difference would be much larger. MAE (mean absolute error) is fully robust to outliers but produces a different fit (median of residuals, not mean).

• Segmentation — model the high-value segment separately. Train one model on orders < $500, another on orders > $500, and route predictions by segment. This preserves the signal from high-value customers and gives each segment a model suited to its scale.

✓ Step 3 complete. Four treatment strategies evaluated: remove (for confirmed errors), cap/winsorize (preserve the row), log transform (compress the scale), keep + robust model (when the outlier carries signal).

Treatment Decision Guide

The choice depends on the cause of the outlier and the business cost of misclassification:

The first question is always causal: is this a data error? If yes, remove. If the scale is inherently exponential (revenue, counts, populations), use a log transform. If the outlier is a real and important observation, keep it and use a robust model. Otherwise, winsorize to cap the value without losing the row.

Comparison of Treatment Strategies

When It Works and When It Doesn't

Univariate detection methods (IQR, z-score) work well when outliers are extreme on a single feature and the dataset is large enough (>20 samples) for quartiles and standard deviations to be stable. They fail when:

• The outlier is multivariate (normal on each feature individually but anomalous in combination)
• The dataset is tiny (<10 samples) — even robust quartile estimates can be unreliable
• The data is heavily skewed — the IQR's 1.5× factor flags too many or too few points depending on tail direction

Hyperparameter Sensitivity

The IQR method's 1.5× factor and the z-score's 3.0 threshold are conventions, not laws. Different values give different outlier counts.

On a single extreme outlier, the IQR factor does not change the verdict — customer 7 is so far above the cluster that any reasonable factor flags it. The factor matters most on borderline cases (e.g., a value 1.6× above Q3, flagged by 1.5 but not by 2.0).

Z-threshold sensitivity is sharp here because customer 7's z is exactly 3.0 — right at the convention boundary. Use a strict threshold only when false positives are costly; loosen to 2.5 when missing an outlier is the bigger risk.

Related Concepts

Backward: the previous post in this section, Feature Engineering: Missing Values and Outliers, covers the overlap between outlier detection and missing-value imputation — extreme missingness patterns can themselves indicate outliers. The IQR method in this post uses the same percentile-based thinking as the quartile discussion there.

Forward: the next topic in this section, Categorical Encoding, deals with a different feature problem — converting non-numeric categories to numbers. Outlier handling operates on numeric columns; encoding produces numeric columns. Together they cover the bulk of feature engineering on real data. Beyond this section, Isolation Forest reappears in Anomaly Detection (unsupervised learning) as a general-purpose tool for unlabeled outlier detection.

Honest Limitations

• With fewer than ~20 samples, IQR and z-score are unstable. A single outlier can shift quartiles and inflate std, masking itself. Use a robust z-score (median and MAD) or skip the test and inspect by hand.
• The 1.5× IQR factor and |z| > 3 threshold are conventions. They were chosen for normal-like distributions. On heavily skewed data, they flag too many or too few points. Examine flagged cases rather than trusting the rule blindly.
• Univariate methods miss multivariate outliers. A point can have every feature in a normal range but still be anomalous in feature space (e.g., 4800 order value with 2 items is more suspicious than 4800 with 50 items). Use Isolation Forest or a domain-specific rule for multivariate cases.

Test Your Understanding

1. Conceptual — Why does a single extreme outlier change the slope of an OLS regression line so much? What property of the loss function causes this?
2. Applied — On the anchor (excluding customer 7), the 9 remaining order_value values have mean ≈ 91.6 and std ≈ 10.7. What is the z-score for 4800 computed without customer 7 in the mean and std? Why is this so different from the z=3.00 you get when customer 7 is included?
3. Applied — Using the anchor, compute the IQR upper fence with a factor of 2.0 instead of 1.5. Is customer 7 still flagged? What does this tell you about the factor's role?
4. Edge case — A revenue dataset has a long right tail: most orders under $200, a few at $2000, and a handful at $50,000+. The business cares about forecasting the high-value segment. Which treatment is appropriate and why? What would the model miss if you used log transform alone?
5. Edge case — A sensor records temperature every minute. One reading is 999°C — clearly a sensor glitch, not a real reading. Should you remove it, cap it, or log-transform the data? What is the correct order of operations?