Distributions in EDA
Know the shape you're really modeling
Histograms, density plots, Q-Q plots and the empirical vs theoretical shape tell you whether normal assumptions hold — and what transform makes skewed data behave.
What you will learn
- Compare shape via histogram and KDE
- Read a Q-Q plot against normality
- Choose log/sqrt/box-cox transforms for skewed data
What is this?
Distribution inspection is where statistics meets exploration: histogram bins and KDE curves describe shape, while Q-Q plots compare empirical quantiles against a theoretical line — straight means 'fits'. Skewness measures asymmetry; kurtosis flags tails. The payoff is choosing the right transform (log, sqrt, Box-Cox) so downstream tests and models don't inherit the skew.
Why do you need it?
- t-tests and regressions assume normal-ish residuals; skewed inputs silently violate them.
- Log-transforming money or latency columns makes means-meaningful again.
- A Q-Q straight-line check is the 10-second normality verdict.
- Heavy tails are the real signal for churn and revenue — shape awareness drives the analysis.
Mental model
A tailor measuring fit against a mannequin
Distribution fitting
The tailor compares each client's measurements to the mannequin's shape; big deviations mean the client isn't a standard fit and the cut must adapt. Q-Q plots measure how far your data deviates from the 'normal mannequin' — and when to stop forcing it.
How it works
- df.hist(bins=...) visualizes shape; .plot(kind='kde') smooths it.
- Q-Q plot: scipy.stats.probplot aligns sample quantiles against theoretical normal; deviation from the line shows where shape diverges.
- skew() and kurtosis() quantify asymmetry and tail heaviness.
- Right-skew: log or sqrt; wait, choose by severity and zeros count.
- Box-Cox (scipy.stats.boxcox) automates optimal power transforms for positive data.
- Compare transformed histograms to confirm the shape improved before modeling.
Shape → diagnose → transform.
Examples
Straighten a skew in three calls
import numpy as np, pandas as pdfrom scipy import statsx = pd.Series(np.random.default_rng(9).lognormal(2, 1.0, 500))print('skew raw :', round(x.skew(), 2))lx = np.log(x)print('skew log :', round(pd.Series(lx).skew(), 2))_, p_shapiro = stats.shapiro(np.clip(pd.to_numeric(lx, errors='coerce').dropna(), -10, 10))print('log shapiro closer to normal than raw')The raw skew plummets after the log transform — the classic heavy-tail repair before modeling.
In production
- A pricing model box-cox transforms 'order value' before regression so residuals normalize.
- An ops analyst logs p99 latency before any ARIMA-style forecast.
- A churn model keeps heavy-tail spend but reports median-based features to avoid whale dominance.
Best practices
- Inspect the histogram before any transform; let the data argue for its own shape.
- Log is for positive, right-skewed data; sqrt for mild skew; Box-Cox when you want automation.
- Re-examine with a Q-Q plot after transforming — verify the repair landed.
- Document the transform with its inverse so models can predict on the original scale.
Common mistakes
- Forcing a normal-ish shape on data that is naturally heavy-tailed and won't comply.
- Log-transforming data with zeros and losing them.
- Interpreting skew() alone without the histogram's context.
- Fitting models on transformed data then quoting predictions in transform units.
Debugging tips
- A Q-Q line that bends at the tail → heavy tails; consider robust tools instead of more transforms.
- A log transform creates a cliff at zero → add 1 or use alternatives (inverse hyperbolic sine).
- skew near 0 but histogram still odd → bimodal, not skewed; segment instead of transform.
- Box-Cox needing strictly-positive values → shift or filter negatives first.
Interview checkpoint
Quick quiz
A Q-Q plot that deviates mainly at both ends suggests:
Log transforms are safe on columns that contain zeros.