Loss Functions and Gradient Steps
Trees as gradient descent steps
Boosting is gradient descent on trees: every new tree is a step in function space, fitted to negative gradient of the loss. Choosing the loss changes the residuals and the whole bias of the model.
What you will learn
- Derive residuals as negative gradients for common losses
- Choose loss by prediction target
- Adjust residuals (Huber) for robustness
What is this?
Boosting minimizes a loss L(y, F) via function-space gradient descent. The residual the next tree fits equals -dL/dF at the current prediction: for squared error that's y - F; for binomial deviance it's a log-odds-based pseudo-residual; for Poisson it's y/exp(F) - 1. Change the loss, change what each tree 'wants' — quantile and Poisson losses make the same loop target quantiles and counts.
Why do you need it?
- The residual definition IS the model bias; pick by target, not habit.
- Explains why tuning trees-without-thinking mis-sets loss on counts/outliers.
- Logistic vs squared → different calibration behavior that matters in production.
- Every boosting framework (XGB/LGBM/CatBoost) is this loop re-implemented fast.
Mental model
A hiker correcting a course each day vs plotting the whole route once
Gradient steps vs one-shot fit
Each day (tree) you measure where the trail is vs where the summit-line is, turn a small angle (learning rate); after enough days the trail converges — step size and hill shape are everything.
How it works
- Choose loss: squared (regression), deviance/log-loss (binary), poisson (counts), quantile (tails).
- Initialize F_0 = minimizer of loss for constants.
- Compute pseudo-residual r_i = -dL/dF(F_{m-1}(x_i)).
- Fit a regression tree to r; leaf values refine via one Newton step approximation.
- Update F_m = F_{m-1} + lr * tree; read model in the loss's own scale (probabilities/quantiles).
Gradient step per loss.
Examples
Three losses, three residual formulas
import numpy as npfrom sklearn.ensemble import GradientBoostingRegressorfrom sklearn.model_selection import cross_val_score rng = np.random.RandomState(0)X = rng.uniform(0, 2, (800, 1))y_noisy = 2 * X[:, 0] + rng.normal(0, 0.2, 800)y_count = rng.poisson(1.5 * (1 + X[:, 0])) sq = GradientBoostingRegressor(loss="squared_error", n_estimators=120, random_state=0)poi = GradientBoostingRegressor(loss="poisson", n_estimators=120, random_state=0)from sklearn.metrics import mean_squared_errorm1 = sq.fit(X, y_count)print("Poisson target fit with squared loss MSE:", round(mean_squared_error(y_count, m1.predict(X)), 2))m2 = poi.fit(X, y_count + 1e-6)print("Poisson target fit with poisson loss RMSE:", round(np.sqrt(mean_squared_error(y_count, m2.predict(X))), 2))Count targets shaped badly under squared loss; the matching loss gives residuals the right prior and the tree curve the right scale.
In production
- Counts (page views, claims) -> loss='poisson' keeps predictions non-negative; squared loss leaks negatives.
- Tail risk (quantiles) -> loss='quantile' with alpha set; published for latency budgets.
- Fraud probability -> binom deviance then calibrate; don't square the labels.
Best practices
- Pick the loss by target semantics, not default; validate Poisson/quantile with the domain metric (deviance/quantile score), not just RMSE; if outliers dominate squared loss, try Huber 90/95 for the robustness sweet spot.
Common mistakes
- Squared loss on counts (negatives, skew); comparing losses by a metric they don't optimize; pushing quantile/poisson predictions through the wrong link when serving (they're expected values already); forgetting to re-fit residuals-scale charts.
Debugging tips
- Negative predictions leak through squared loss on counts -> switch to Poisson/Newton.
- AUC fine but probabilities carry a bias -> deviance/logistic loss + isotonic calibration to close.
- One tail dominating residuals -> raise Huber alpha or switch quantile.
Interview checkpoint
Quick quiz
For squared error, the next tree's target is:
Poisson loss is the natural choice for: