Advanced 14 min read XGBoost

Imbalance and Categorical Features in XGBoost

Two papercuts, two knobs

scale_pos_weight re-centers the loss for skewed targets; enable_categorical=true + fold hints let XGBoost split native categoricals — the two knobs that stop common tabular papercuts.

scale_pos_weightclass imbalancecategorical featuresenable_categoricalfold splitting

Prerequisites: Early Stopping and Eval Sets

Before we start

What you will learn

Set scale_pos_weight from the class ratio
Enable native categorical handling with fold hints
Decide when to pre-encode instead

Introduction

What is this?

For imbalanced targets set scale_pos_weight ≈ negative/positive count (or use sample weights) so the gradient doesn't drown the rare class. For categorical columns, XGBoost 1.6+ supports native categorical data via enable_categorical=True and a fold-splitting hint on the DMatrix; otherwise one-hot/binary-encode first. Both knobs change results more than 'post-hoc threshold tweaks' alone.

Motivation

Why do you need it?

Rare-class drowning is the #1 silent issue on fraud/CLV/claim tables.
scale_pos_weight moves the model's prior, not just the decision threshold — fixes earlier in the pipeline.
Native categoricals tree-split better than one-hot on high-cardinality columns (saves splits + memory).
Interviews expect you to know WEIGHT vs THRESHOLD — two different levers.

Think about it

Mental model

Imagine

Training a lifeguard in a lake with very few swimmers

Think of

scale_pos_weight

If the rare 'swimmer in trouble' class is 1-in-1000, the model learns 'all safe' unless tank times those rare catches. Weighting persuades the trainer to care about the scarce-but-critical case.

Mechanics

How it works

Ratio: ratio = n_negative/n_positive; set scale_pos_weight ≈ ratio (start mid-ranges).
Or pass sample_weight per row for fine-grained priors.
eval_metric 'auc-pr' / classification reports — not plain accuracy — on imbalanced sets.
Categorical: mark column dtype 'category'; XGBoost hosts fold-of-categories split (no dummy explosion).
Fallback: one-hot or target-encode small/detailed categoricals before fit when needed.

Imbalance + categorical playbook.

Rare classscale_pos_weight

Score rightauc-pr not accuracy

Categoricalscategory dtype + fold hint

Encode fallbackbinary/high-cardinality handling

Hands-on

Examples

Weighted vs not

import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import average_precision_score
 
X, y = make_classification(n_samples=5000, n_features=18, n_informative=6,
                           n_redundant=0, weights=[0.97, 0.03], random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, stratify=y, random_state=1)
neg, pos = (ytr == 0).sum(), (ytr == 1).sum()
for w in [1, neg / pos]:
    m = xgb.XGBClassifier(n_estimators=150, scale_pos_weight=w, random_state=0,
                          eval_metric="aucpr")
    m.fit(Xtr, ytr)
    pr = average_precision_score(yte, m.predict_proba(Xte)[:, 1])
    print(f"scale_pos_weight={w:>4.1f} avg-precision {pr:.4f}")

Weighting toward the majority:minority ratio rescues the rare-class precision — measured on average precision (PR), not accuracy.

Real world

In production

Fraud scoring: scale_pos_weight tuned around the business-approved precision cutoff.
High-cardinality sector/store columns: fold-split categories without exploding the matrix.
Compliance cuts: decide early whether prevalence shifts (drift) need weight re-tuning, listed in the card.

Guidelines

Best practices

Set weights from true positive cost, not just the ratio; score imbalanced builds with auc-pr/PR-curve, never raw accuracy; mark categories as 'category' dtype BEFORE building DMatrix; verify fold hint by comparing to one-hot baseline on a sample.

Watch out

Common mistakes

Over-weighting until you memorize the rare class; accuracy-gazing on a 3% positive set; one-hotting hundreds of categories into a bloated matrix; setting enable_categorical without the dtype contract.

When things break

Debugging tips

Precision stays ~0 regardless of weight => eval metric wrong/lr too high; check the PR curve, not totals.
Huge feature columns after category folds => too many categories; target-encode or bucket first.
Fold hint slower than one-hot on tiny data => native path has overhead; keep one-hot for small sets.

Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2

scale_pos_weight influences processing at:

Native categorical splits (XGB 1.6+) require:

Go deeper

Official documentation

Categorical feature support (XGBoost)

docs

scale_pos_weight param docs