Beginner 10 min read Decision Trees

Decision Trees Overview

A flowchart the model learns

Decision trees recursively split the data on the most informative feature threshold, yielding interpretable if-then rules — the readable base on which forests and boosting are built.

decision treesplitsrecursive partitioningif-then rulesinterpretablebias variance

Prerequisites: ML Fundamentals: Overfitting

Before we start

What you will learn

Trace a tree's recursive partition down to leaves
Explain feature-importance intuition from split usage
Know what makes trees interpretable AND what they sacrifice

Introduction

What is this?

A decision tree recursively splits the feature space: at each node pick the feature and threshold that best separates the target, and repeat into leaves that are pure enough. Prediction = follow the learned if-then path. Trees are non-parametric, scale-free (no scaling needed), handle mixed feature types, and produce a flowchart humans audit. The catch: a deep tree memorizes noise if you don't cap it.

Motivation

Why do you need it?

Interpretability: a tree can be printed and explained to a stakeholder line by line.
They're the atomic unit of Random Forest and Gradient Boosting — master this and ensembles unfold.
Scale indifference: no feature-standardization required, mixed types welcome.
Threshold splits capture non-linear and interacting structure automatically.

Think about it

Mental model

Imagine

A doctor's diagnostic flowchart

Think of

A learned decision tree

Fever? > 38 deg -> then cough? -> then ... Each question (split) narrows the possible answers until a diagnosis (leaf) claims the patient.

Mechanics

How it works

Start with the full dataset at the root.
For each candidate split (feature, threshold), score the partitions by impurity decrease (Gini/entropy).
Pick the winning split; partition rows into child nodes; recurse.
Stop or prune: max_depth, min_samples_split, min_samples_leaf, or purity.
Predict by routing a new row down the path to its leaf; class = leaf majority / regression = leaf mean.

Recursive partitioning.

Rootall data, best split

Childsubset, next best split

Leafpure enough — predict

Pathfollow thresholds downward

Hands-on

Examples

A tiny traceable tree

import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
 
d = load_iris()
Xtr, Xte, ytr, yte = train_test_split(d.data, d.target, test_size=0.3,
                                      stratify=d.target, random_state=1)
m = DecisionTreeClassifier(max_depth=3, random_state=0).fit(Xtr, ytr)
print("depth:", m.get_depth(), "leaves:", m.get_n_leaves())
print("test acc:", round(m.score(Xte, yte), 3))
print("feature importances:", dict(zip(d.feature_names, m.feature_importances_.round(3))))

A depth-3 tree prints as a tiny flowchart; importances reflect which splits earned their place at the top of the recursion.

Real world

In production

Credit rules: a shallow tree IS the written underwriting policy a regulator can read.
Triage gates: interpretable first-pass rules before a large model decides.
Feature audit: tree rankings are a fast, common proxy before switching to forests.

Guidelines

Best practices

Cap complexity (depth/leaf size) FIRST; validate with CV; prefer y in balanced-feature semantics (tree splits are scale-free); print/save the tree for the audit trail.

Watch out

Common mistakes

Letting trees grow to full depth (overfit on default settings); reading importances as causal; pushing mixed problem types into one tree expecting robustness (single trees wobble); using a deep single tree where a forest does the job

When things break

Debugging tips

Save and plot the tree for 2-3 validation misfits — the failing path usually reads like a data issue.
Train accuracy 100% + validation dropping = depth cap/leaf-size fines needed.
Importance concentrating on one engineered feature => check that feature's data lineage.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

Prediction in a tree routes by:

A fully grown (unpruned) tree typically:

Go deeper

Official documentation

Decision Trees (sklearn)

docs

Decision tree learning | Wikipedia