EDA
Intermediate 12 min read EDA

Automating EDA

The audit, generated on demand

Repeatable profiles can be generated: a reusable profiling module (pandas-profiling, sweetviz, or your own audit function) turns the EDA ritual into one call and one report.

pandas-profilingsweetvizautomationreportabstract
Prerequisites: EDA Workflow
Before we start

What you will learn

  • Run an auto-profile and read its sections
  • Build a minimal reusable audit function
  • Know when automation replaces vs supplements judgment
Introduction

What is this?

Automated EDA packages (pandas-profiling/ydata-profiling, sweetviz) generate a full report — distributions, missingness, correlations, duplicated, alerts — with one call. Lightweight personal versions (a 20-line audit function) do the 80% without a heavy dependency. Automation standardizes the audit so the human spends energy on interpretation, which tools can't do.

Motivation

Why do you need it?

  • Ritual code becomes boilerplate; one function replaces 30 repeated cells.
  • Fresh data lands weekly; the same audit should run every time without copying cells.
  • Teams agree on one profile shape, making hand-off comparisons meaningful.
  • Alerts derived from automated profiles (missing-rate > threshold) run in CI.
Think about it

Mental model

Imagine

A blood-test panel at a clinic

Think of

Automated EDA report

The panel runs the same standard markers every time, giving the lab technician (and then the doctor) a fixed set of values to interpret. Automation doesn't decide the diagnosis — it makes the evidence fast and consistent.

Mechanics

How it works

  • profile = ProfileReport(df) returns a full HTML/summary (ydata-profiling).
  • Sections: overview, variables, interactions, correlations, missing, sample, alerts.
  • sweetviz compare(df_train, df_test) highlights train-vs-test distribution drift.
  • A custom audit() reuses the describe/nunique/isna/dtype block with thresholds.
  • Reports export to HTML and can be pasted into docs or CI pipelines.

Auto-profile → human interpretation.

DataFrame
ProfileReport
HTML report
Human reads
Feed pipeline
Hands-on

Examples

A 15-line audit you can always run

A 15-line audit you can always run
import pandas as pd
 
def audit(df, max_levels=20):
    n = len(df)
    rows = []
    for c in df.columns:
        s = df[c]
        rows.append({
            "col": c, "dtype": str(s.dtype),
            "null_rate": round(s.isna().mean(), 3),
            "nunique": s.nunique(),
            "constant": s.nunique() == 1 or (len(s) and s.nunique() <= 1),
            "levels_ok": True
        })
    return pd.DataFrame(rows)
 
print(audit(pd.DataFrame({"a": [1,2,None], "b": ["x","x","x"]})))

A dependency-free audit that ranks every column's type, null rate and uniqueness — the pipeline front-end for nightly data checks, no extra library required.

Real world

In production

  • A nightly job runs audit() against thresholds and pages when a column's null rate doubles.
  • A team ships ydata-profiling HTML into the repo's docs for each dataset version.
  • Eng-engineering uses sweetviz.compare(train, test) in CI as a lightweight drift gate before promoting features.
Guidelines

Best practices

  • Standardize ONE profile tool across the team.
  • Bound report size: sample huge frames down before profiling.
  • Treat alerts as review prompts, not failures — automation points, humans decide.
Watch out

Common mistakes

  • Believing a generated report IS the EDA — interpretation is the human half.
  • Profiling 50M rows directly and timing out; sample first.
  • Trusting alerts without reading the drill-down (a null-rate alert in an unmixed column context).
When things break

Debugging tips

  • If the profile shows NaN rates you don't recognize, profile a subset before the full frame to isolate the column.
  • Compare two reports (train vs test) for the same columns to catch drift at the source.
  • Version profiled data with a hash so 'did the data change or the code?' is answerable.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

ydata-profiling's ProfileReport(df) is best described as:

2

Generated EDA reports remove the need for human interpretation.

Go deeper

Official documentation

ydata-profiling docs
docs
sweetviz docs
docs