Features, Labels & Datasets
The material models are made of
Features are what the model sees; labels are the truth you train against. Dataset quality — coverage, labeling consistency, class balance — usually matters more than the model architecture.
What you will learn
- Distinguish features, labels and dataset structure
- Spot labeling noise and class imbalance
- Explain why dataset quality often beats model choice
What is this?
A training dataset is rows of features (inputs) and labels (answers). Features are the numbers, tokens or embeddings the model reads; labels are the ground-truth outcomes — 'spam' or 'not spam', a price, a category. ML is only as good as this material: noisy labels teach errors, biased coverage teaches bias, and imbalanced classes teach the easy majority.
Why do you need it?
- Feature engineering decides what information is even visible to the model.
- Labeling noise caps accuracy regardless of architecture — 'garbage in' really is fatal.
- Class imbalance makes 'always predict majority' the naive winner and hides real degradation.
- The highest-leverage move in ML is almost always better data, not a bigger model.
Mental model
Teaching with a bad textbook
Training on low-quality labels
A brilliant student (a big model) with a typo-riddled textbook still gets the answers wrong. The textbook — the labeled data — is the ceiling.
How it works
- Features encode each example: a text's embedding vector, a row's numeric fields, a row's categorical encodings.
- Labels are assigned by humans or heuristics; each carries your definition of 'correct'.
- The dataset splits into training (model adapts), validation (tuning) and test (final estimate) — kept strictly apart.
- Redundant/near-duplicate and misleading rows are cleaned so the model learns the signal, not noise.
- Balance or reweight classes so minority classes are learned rather than ignored.
Rows of features, one label column.
In production
- A churn model uses features (last login, plan, tickets) and labels (churned yes/no) from a 12-month window, split by time.
- A content classifier flags low-confidence labels for human review instead of training on garbage.
- A fraud dataset down-samples the majority class so the 2% fraud signal is actually learnable.
Best practices
- Capture label provenance and confidence; store the labeling guideline with the data.
- Estimate label noise with double-labeling on a sample and reconcile disagreements.
- Split by time or group for realistic evaluation, never by random row when time matters.
- Baseline the majority class so minority performance is judged honestly.
Common mistakes
- Letting row-level leakage spread features between train and test (normalization fit on the full set).
- Training on imbalanced data without measuring per-class behavior.
- Treating duplicate rows as independent samples, inflating confidence spuriously.
Common issues
Model 'learns' the label column magnitudeA feature that is a transformed version of the label leaked in
Profile features for label-derived columns and remove leaky ones.
Accuracy high, but minority class is uselessImbalanced data with accuracy as the metric
Report precision/recall per class and rebalance or reweight training.
Human judges disagree on the labelsAmbiguous labeling guideline
Tighten the guideline, double-label a sample, and measure agreement.
Debugging tips
- Model samples every example as the majority class — check class balance first.
- Great train, worse test — look for leakage or duplicated rows shared across splits.
- A feature that looks predictive alone might be post-processing the label; check for temporal leaks.
Interview checkpoint
Hands-on lab
- Generate a small synthetic labeled dataset
- Detect class imbalance and duplicate rows
- Verify train/test splits don't share leakage signals
Profile a Dataset for Leaks
20 min · beginner
Quick quiz
What does a label represent?
Random row splits are always a safe choice for training/test.