Intermediate 14 min read Decision Trees

How Splits Work: Gini, Entropy and Information Gain

The math of 'which split?', every node down

Every node picks the (feature, threshold) that maximizes impurity decrease — Gini and entropy measure impurity, information gain is the weighted decrease that drives CART and ID3 alike.

ginientropyinformation gainimpuritysplitCARTclassification tree

Prerequisites: Decision Trees Overview

Before we start

What you will learn

Compute Gini impurity and entropy for a node
Calculate information gain as the weighted impurity decrease
Explain why a perfect splitter can still pick a pointless feature

Introduction

What is this?

Tree algorithms score candidate splits by impurity reduction. Gini impurity = 1 - Σ p_k^2 (probability of mislabeling a random row under weighted guessing); entropy = -Σ p_k log2 p_k. Information gain = parent impurity minus the size-weighted average of child impurities. Continuous features: sort values, try each mid-point threshold, pick the max-gain (feature, threshold) at the node.

Motivation

Why do you need it?

Splitting rules ARE the learning: nothing else decides the tree's shape.
Gini vs entropy differences explain fractional performance gaps between configurations.
Gain-based reasoning extends verbatim to boosting step sizes and feature importance.
Interviewers love the 'why Gini < 1 - sum p^2 < entropy' nuance.

Think about it

Mental model

Imagine

Splitting a mixed deck of cards into pure piles

Think of

Impurity-based splits

A good division leaves each pile nearly single-suited (low impurity); a bad one strands suits mixed in each pile (high impurity). The tree always tries the division that purifies the most.

Mechanics

How it works

Compute parent impurity: Gini = 1 - Σp_k^2 or Entropy = -Σp_k log2(p_k).
For each feature + candidate threshold: split rows; weight child impurities by size.
Gain = impurity_parent - (nL/n)imp_L - (nR/n)imp_R.
Choose the split with the highest gain (a.k.a. lowest weighted child impurity).
Repeat per node until stopping criteria; ties broken by convention (first feature).

Gain formula pictorially.

Parentimpurity I

Split L/Rweighted child impurities

GainI - weighted(I_L,I_R)

Choosemax-gain (feature, threshold)

Hands-on

Examples

Gini vs entropy on one split

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
 
d = load_iris()
X, y = d.data[:, 0:1], d.target
 
def gini(p):
    return 1 - np.sum((np.bincount(p) / len(p)) ** 2)
 
def ent(p):
    c = np.bincount(p) / len(p)
    return -np.sum(c * np.log2(c + 1e-12))
 
print("root Gini:", round(gini(y), 4), " entropy:", round(ent(y), 4))
 
for crit in ["gini", "entropy"]:
    t = DecisionTreeClassifier(criterion=crit, max_depth=1, random_state=0).fit(X, y)
    print(f"{crit:>6}: best split on feature 0 at x > {round(t.tree_.threshold[0], 2)}"
          f"  gain {round(t.tree_.weighted_n_node_samples[0] / len(y) * (t.tree_.impurity[0] - t.tree_.impurity[1]*t.tree_.weighted_n_node_samples[1]/t.tree_.weighted_n_node_samples[0] - t.tree_.impurity[2]*t.tree_.weighted_n_node_samples[2]/t.tree_.weighted_n_node_samples[0]), 3)}")

Both criteria choose the same threshold near 5.5 because purity ordering agrees even though the two impurity scales differ — the split, not the scale, is what one tree line means.

Real world

In production

Cap tuning: gain curves show where a feature stops carrying signal (deep nodes, tiny gains -> prune).
Feature screening: cumulative gain of top features shortcuts a RF importance chart.
Explaining to review: 'Gini-favored split' is an auditable sentence in a model card.

Guidelines

Best practices

Default to Gini (faster) unless entropy empirically wins; use the gain CURVE for pruning decisions, not a single number; remember both criteria usually pick the same qualitative splits.

Watch out

Common mistakes

Perceiving 'higher gain' as 'more important feature' globally (nodes deep down see few rows); expecting purity-based splits to know about future depth (greedy optimization is local); benchmarking empty on continuous vs categorical handling.

When things break

Debugging tips

Two features give equal gain repeatedly => either collinear; check with an importance plot.
Gain tiny at high depth => signal spent; prune there.
Criterion change flips tree shape => data has near-tied splits; check stability.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

Gini impurity at a pure node equals:

Information gain is:

Go deeper

Official documentation

sklearn tree criterion docs

docs

Information gain in decision trees | Wikipedia