GroupBy and Aggregation
The split-apply-combine engine
split-apply-combine turns 'group by region, summarize revenue' into one pipeline: group keys, apply aggregations or transforms, and recombine — the pivot of all tabular analytics.
What you will learn
- Write groupby-key fields and aggregations
- Use agg() for per-column aggregate mixes
- Contrast agg vs transform vs apply
What is this?
df.groupby(key)[cols].agg('sum') implements split-apply-combine: split the frame into groups by key, apply per-group operations, and combine results. agg() returns one row per group with named summaries; transform() broadcasts group stats back to each row (e.g., row - group mean); apply() runs arbitrary functions per group.
Why do you need it?
- Nearly every business metric is a grouped aggregation: revenue by region, churn by tier.
- It replaces dozens of hand-written loops with one readable pipeline.
- transform powers 'deviation from group average' feature engineering.
- pivot_table is the same machinery laid out as a matrix.
Mental model
Sorting mail by postal code, reading each pile's total weight
Groupby + aggregate
You split the sacks by zip (groupby), weigh each pile (agg), and report one number per pile — split, apply, combine. A friendly mailroom does this every day.
How it works
- df.groupby(['region'])['revenue'] isolates a column to aggregate per region.
- .agg({'revenue': 'sum', 'orders': 'mean'}) applies different stats per column.
- Named-agg keeps readable result columns: .agg(total=('revenue','sum')).
- .transform('mean') returns the group mean broadcast to every original row.
- .pivot_table(index='region', columns='tier', values='revenue', aggfunc='sum') reshapes into a 2D grid.
Split-apply-combine.
Examples
Groupby quality kit
import pandas as pddf = pd.DataFrame({ "region": ["eu","eu","us","us","us"], "orders": [10, 5, 8, 4, 6], "rev": [100, 60, 90, 70, 40],})print(df.groupby("region")["rev"].sum()) # revenue per regionprint(df.groupby("region").agg(total=("rev","sum"), avg_order=("orders","mean")))df["rev_from_mean"] = df["rev"] - df.groupby("region")["rev"].transform("mean")print(df[["region","rev","rev_from_mean"]])A grouped sum, a named-aggregation summary, and a transform that subtracts each region's mean — a feature-pattern you'll reuse constantly. Each answers one business question without loops.
In production
- A revenue dashboard computes 'daily revenue by region' from a page of groupby pipelines.
- A feature store columns 'purchase-amount deviation from user average' via groupby.transform — the classic ratio features.
- A quality report groups by table/model to compute row and null counts per artifact.
Best practices
- Use agg() with explicit column→function mappings; named-agg reads best in review.
- Prefer transform for group-broadcast statistics (keeps row alignment).
- Set as_index=False or .reset_index() when the group key shouldn't become the index.
Common mistakes
- Chaining df.groupby('r')['c'].sum() and then trying to reset the now-indexed result twice.
- Confusing agg (one row per group) with transform (one row per original row).
- Grouping by an unnormalized key so 'EU' and 'eu' split silently.
Debugging tips
- df.groupby('region')['rev'].agg(['sum','count','mean']) dumps a diagnostic table in one call.
- For missing groups check value_counts() on the key — empty groupby output usually means unmatched keys.
- transform results can be verified with a hand-computed group mean on one group.
Interview checkpoint
Hands-on lab
- Compute grouped aggregations with named results
- Build a pivot table
- Engineer a transform-based ratio feature
GroupBy to ANSWERS
30 min · intermediate
Quick quiz
df.groupby('region')['rev'].sum() returns:
.transform('mean') returns groups' means aligned to the original rows.