Exploratory Data Analysis
Intermediate 12 min read Exploratory Data Analysis

Correlation Matrices

Every pairwise link in one glance

One heatmap surfaces every pairwise relationship at once — the fast scan for redundancy, confounders and candidate features before any regression.

correlationheatmapmatrixmulticollinearitypearsonredundancy
Prerequisites: Bivariate Analysis
Before we start

What you will learn

  • Compute and read a correlation matrix
  • Spot redundant and confounded pairs
  • Guard correlations against outliers and scales
Introduction

What is this?

df.corr() computes pairwise Pearson correlation for numeric columns; a hedged heatmap renders it. Reading it: high |r| between two features means redundancy (multicollinearity); high |r| with the target is a feature candidate; small r everywhere means weak linear links. The matrix is a triage map for modeling.

Motivation

Why do you need it?

  • Multicollinear features destabilize regression coefficients.
  • A heatmap exposes confounds across dozens of features in one look.
  • Feature selection starts with 'which variables actually move with the target?'.
  • It flags nonsensical-but-correlated pairs worth investigating.
Think about it

Mental model

Imagine

A social network's friendship map

Think of

A correlation matrix

Thick lines between clusters reveal groups that always appear together — redundant features — and the bridge nodes connect otherwise separated clusters, like would-be confounders. A heatmap is the social graph of your columns.

Mechanics

How it works

  • df.corr() defaults to Pearson on numeric columns; min_periods controls sparsity.
  • A heatmap (seaborn.heatmap) visualizes the matrix with symmetrical color scale.
  • Cluster ordered columns to make structure pop (hierarchical clustering).
  • Robust checks: Spearman (df.corr(method='spearman')) resists outliers.
  • Threshold scans: list feature pairs with |r| > 0.8 for redundancy review.
  • Always re-verify a highlighted correlation with the scatter — r describes slope, not shape.

Matrix → actions.

Compute
Render
Read
Verify
Act
Hands-on

Examples

Scan, verify, keep the honest set

Scan, verify, keep the honest set
import numpy as np, pandas as pd
rng = np.random.default_rng(10)
df = pd.DataFrame({"a": rng.normal(size=100), "b": rng.normal(size=100),
                   "c": rng.normal(size=100)})
df["a_plus"] = df.a + rng.normal(0, 0.1, 100)   # near-duplicate of a
df["y"] = 2 * df.a + 0.5 * df.b + rng.normal(0, 0.2, 100)
cm = df.corr().round(2)
print(cm["y"].sort_values(ascending=False))
print("dup pair a/a_plus:", round(cm.loc["a", "a_plus"], 2))

The target's column of the matrix ranks feature value; the a/a_plus cell exposes redundancy before modeling.

Real world

In production

  • A feature store drops one of each |r|>0.95 pair before training, stabilizing coefficients.
  • An experiment QA matrix flags that 'time on page' and 'session length' are the same signal.
  • A churn model keeps only target-correlated features within each redundancy cluster.
Guidelines

Best practices

  • Round the matrix for readability; format annotations on the heatmap.
  • Verify top pairs with scatter plots — r hides nonlinear and outlier-driven shapes.
  • Use Spearman for skewed money/latency features before trusting Pearson.
  • Record redundant-pair findings in the EDA report for the modeling phase.
Watch out

Common mistakes

  • Trusting a high Pearson r from three outliers that drag the line.
  • Ignoring pairs that are perfectly correlated because they're the same column re-expressed.
  • Reading causality into targeted correlations.
  • Forgetting that corr() only sees numeric columns — and NaN pairs are skipped per pair.
When things break

Debugging tips

  • A near-1 correlation that surprises → one column is derived from the other; compute the difference.
  • Negative r flip after segmentation → confound; stratify.
  • corr() with many NaN values misleads — min_periods keeps only well-observed pairs.
  • A zero correlation that doesn't match the scatter → the relationship is nonlinear; r is linear-only.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Two features at |r| ≈ 0.95 most likely imply:

2

Pearson correlation is robust to extreme outliers.

Go deeper

Official documentation

pandas: DataFrame.corr
docs