Model Poisoning Attacks
The enemy inside the training window
Poisoning corrupts the model through its data: label-flips and example injections at train time, or backdoors that activate under secret triggers. Defenses are provenance-first: versioned, access-controlled data, anomaly-screened updates, and honesty metrics that don't trust a promising retrain.
What you will learn
- Classify poisoning vectors: data injection, label flip, backdoor
- Secure the data lifecycle against tampering
- Detect poisoned retrains instead of celebrating them
What is this?
Model poisoning manipulates the training data so the resulting model misbehaves in attacker-chosen ways — injected examples, flipped labels, or backdoored triggers — typically introduced through upstream datasets, weak labeling pipelines, or contributors. Defense is provenance and integrity, end to end.
Why do you need it?
- A poisoned retrain doesn't look broken — it looks improved, so it ships faster.
- Upstream data and labeling are the least-controlled inputs in the ML lifecycle.
- Backdoored models pass clean metrics and only fail on trigger inputs.
- It attacks the future: every future release inherits the poisoned base.
Mental model
A smuggled worker joining the assembly line
A poisoned training entry
One worker (one corrupted example) looks like any other; but their instruction manual says 'when I flash the sign, break this part'. The factory passes inspection — the parts fail only when the trigger is shown. You can't inspect the finished product for it; you must control who joins the line.
How it works
- Identify the ingress: raw data dumps, crowd labels, scraped sets, model-contributed data.
- Label-flip: attacker flips a targeted slice's labels (often low-sample classes) to bias decisions.
- Backdoor: attacker injects examples with a secret trigger + desired (wrong) label; trigger-unlabeled acts clean.
- Defense: versioned, signed data snapshots; provenance/PHI review; anomaly screens on additions; validation on canary subsets.
- Detection: outlier example analysis, label-review sampling, and model-level trigger probing.
Poisoned pipeline and its gate.
In production
- A content moderation retrain 'improves' accuracy but secretly downgrades a minority class — traced to a label-flip in that segment's provider dataset.
- A chatbot's new training data includes injection-style examples that the system then repeats verbatim under a phrase prefix — a backdoor pattern.
- A bank's fraud retrain inherits a poisoned realistic-fraud subset from a shared marketplace dataset.
Best practices
- Treat every dataset revision as a deployable artifact: signed, versioned, access-logged.
- Anomaly-screen additions: outlier features, duplicated rows, impossible label/feature combos.
- Keep canary subsets: if behavior changed only on the canary's trigger-region, don't merge.
- Audit labeling vendors and provenance contracts; expiry ground, not trust.
Common mistakes
- Auto-merging a 'promising' dataset without engine-verifying where its rows came from.
- Trusting provenance by file name only — no hash, no signature, no audit.
- A single big retrain, 'improvement' taken at face value, then the backdoor activates.
- No model-behavior probes — clean metrics pass, trigger behavior never tested.
Debugging tips
- Regressors drift on a low-sample segment only — inspect that segment's labels/rows for flips.
- Behavior is great until you mention a keyword — that's a backdoor signature; probe trigger-adjacent patterns.
- Metrics improve but the seems-wrong class wobbles — pull the canary data and re-validate that slice's labels.
Interview checkpoint
Hands-on lab
- Plant a targeted label flip
- Show clean-metric silence on the poisoned retrain
- Catch it by segment-level and trigger analysis
Detect a Poisoned Retrain
35 min · advanced
Quick quiz
A backdoor attack is best described as…
A poisoned retrain usually looks worse on clean metrics.