Filtering and Selection
Ask the DataFrame any question
Filtering is the grammar of questions: 'who, what, where' maps to boolean masks, isin, query and multiple conditions — the skills behind every segment, cohort and targeted analysis.
What you will learn
- Build boolean masks from conditions
- Combine conditions with & | ~ and isin/query
- Select rows and columns together precisely
What is this?
Filtering selects rows by truth: mask = df.sales > 100 yields a boolean Series; df[mask] keeps only True rows. Conditions compose with & (and), | (or), ~ (not); isin() matches sets; query() expresses conditions as strings for long pipelines. Column selection rides along via loc/iloc.
Why do you need it?
- 'Show me enterprise customers with falling usage' is a two-condition filter.
- Comparisons on the whole column at once are fast and readable.
- Segments, cohorts and exclusions are all filters — this is the bread and butter of analysis.
- Composing masks avoids the loop-with-if antipattern entirely.
Mental model
A bouncer at the door with a guest list and checks
Boolean filtering
The bouncer lets in people who pass (on the list AND over 21) OR (VIP). Each condition is a check; the AND/OR logic composes them; the mask is the decision about every person in line — all at once.
How it works
- df['sales'] > 100 returns a boolean Series aligned to the index.
- df[mask] keeps rows where the mask is True; chaining df[mask][['city','sales']] selects columns too.
- Combine conditions with & | ~ inside parentheses: df[(df.plan == 'pro') & (df.sales > 100)].
- df[df.city.isin(['austin','dallas'])] matches a set; df.query("sales > 100 & plan == 'pro'") reads as a string.
- loc takes row labels/conditions and column names together: df.loc[mask, ['city','sales']].
Filter assembly.
Examples
Two-condition question, one line
import pandas as pddf = pd.DataFrame({ "city": ["austin", "houston", "austin", "dallas", "houston"], "plan": ["pro", "basic", "pro", "enterprise", "basic"], "sales": [120, 60, 80, 400, 30],})mask = (df.plan == "pro") & (df.sales >= 80)print(df.loc[mask, ["city", "sales"]])print(df.city.isin(["austin", "houston"]).sum())An AND mask filters to pro plans selling at least 80; isin() counts rows in either city.
In production
- A churn watch filters accounts where usage dropped below baseline AND plan is high-value.
- An audience builder segments users with isin() across a product list for a UI rollback analysis.
- A dashboard query parameterizes its segment via a string built with query().
Best practices
- Parenthesize compound conditions — operator precedence humbles everyone.
- Prefer isin() to repeated == or ; chained comparisons.
- Use query() for long filter strings; keep loc for combined row+column selection.
- Reuse masks as variables so the filter logic is greppable and testable.
Common mistakes
- Using 'and'/'or' instead of '&'/'|' on Series — the most common pandas error.
- Comparing a column to None/NaN with == (use .isna() instead).
- Forgothing parentheses around conditions next to &.
- Filtering by a column name inside query() with a value containing a quote.
Debugging tips
- A 'ValueError: truth value ambiguous' means you used and/or instead of &/|.
- Mask all-False? Print the mask.sum() and the raw comparison against three samples.
- Series index misalignment after a filter — reset_index or use .loc to avoid odd unions.
- When query() errors on a name, check for spaces or reserved words in the column names.
Interview checkpoint
Hands-on lab
- Compose multi-condition boolean masks
- Use isin and query for readable filters
- Verify segment counts against hand-counted truth
Build Report-Ready Segments
25 min · beginner
Quick quiz
df[(df.a > 1) & (df.b < 5)] uses:
Filtering 'never weekends' keeps returning weekends. Likely cause?