EDA Introduction
Understand before you model
Exploratory Data Analysis is the deliberate, hypothesis-forming stage: shape, dtypes, nulls, distributions and relationships get examined BEFORE modeling so bad assumptions never reach the model.
What you will learn
- Run the first-pass audit: shape, dtypes, nulls, head/tail
- Form hypotheses that EDA validates
- Know when EDA is done vs 'never-ending'
What is this?
EDA (Exploratory Data Analysis) is the structured, hypothesis-driven examination of a dataset before modeling: inspect shape and dtypes, quantify missingness, profile distributions, and probe relationships. It's also the safety net — anomalies found here (units, encoding, starvation, duplicates) are bugs that would silently poison the model later.
Why do you need it?
- Models amplify data issues: a unit mixup in EDA becomes a confident wrong prediction in prod.
- Distribution and missingness facts decide which imputation, scaling and encoders to use.
- Hypotheses formed in EDA make modeling deliberate, not lottery-driven.
- Seniority is visible in the rigor of EDA — it separates analysis from guessing.
Mental model
Reading a map and walking the territory before building a house
EDA before modeling
You wouldn't lay foundations on land you haven't walked — slope, water, rock. Data foundations need the same reconnaissance: where's it swampy (missing), where's the cliff (heavy skew)?
How it works
- First pass: df.shape, df.head(), df.dtypes, df.isna().sum(), df.describe(), duplicated().sum().
- Per column: value counts, distribution (hist/KDE), and unit/semantics sanity checks.
- Pairs: correlations, crosstabs, group summaries to map relationships.
- Anomalies get explained: outlier → data bug or real (then flag, don't delete).
- Every pass closes the loop on hypotheses; notes accumulate into the analysis memo.
The EDA loop.
Examples
The 60-second audit
import pandas as pddf = pd.read_csv("sales.csv")print(df.shape)print(df.dtypes)print(df.isna().sum().sort_values(ascending=False).head())print(df.describe(percentiles=[.05, .5, .95]).round(2))Five lines produce the whole first-pass audit: dimensions, types, missingness ranks and percentile-anchored describe. The p95 vs p50 gap on a column is a free outlier/shape warning before plotting anything.
In production
- A new source lands; a data-engineer run prints the audit block before approving it as a feature table.
- A modeling hand-off requires a one-page EDA memo: keys, dtypes, anomalies, sketched distributions.
- A drift victim: monthly EDA cross-checks catch a schema rename that modeling never noticed.
Best practices
- Never let modeling touch data EDA hasn't described.
- Document anomalies as you find them — write them down or they vanish by review time.
- Cap EDA breadth: focus on columns that will actually enter the model.
Common mistakes
- Skipping EDA and shipping a model that treats '0' and 'missing' as identical.
- Deleting 'weird' rows instead of understanding them.
- EDA on the full population while the model only serves a sliver.
Debugging tips
- When a model behaves unexpectedly, rerun the audit block — a dtype or missingness change is the usual reveal.
- Check describe() against domain bounds; a negative age flag is a parse bug, not a modeling problem.
- value_counts of the target: a distribution you never inspected is a silent class-imbalance landmine.
Interview checkpoint
Hands-on lab
- Run a full audit: shape, dtypes, nulls, duplicates
- Spot a planted data bug via describe()
- Write up findings as hypotheses
First-Pass Audit Sprint
25 min · beginner
Quick quiz
The first-pass EDA audit typically prints all EXCEPT:
Outliers found in EDA should always be removed.