Supply Chain Security
Verify everything that flows into training
The ML supply chain reaches from base images and package locks to datasets and pipeline code. Secure it by pinning and signing artifacts, scanning dependencies, and treating every pipeline input as attacker-controlled until verified.
What you will learn
- Inventory the ML supply chain from image to dataset
- Pin, sign, and scan every stage artifact
- Guard pipelines against dependency and template abuse
What is this?
ML supply-chain security covers the full artifact flow — base images, pip/conda dependencies, MLOps templates, datasets, and model artifacts. Controls: pinning by digest/hash, signed artifacts, vulnerability scanning, SBOM generation, and least-privilege pipeline credentials.
Why do you need it?
- Second-hand code and data dominate ML dependency graphs.
- A compromised dependency ships a backdoored model without touching your model code.
- Untracked dataset changes poison models silently.
- Pipeline secrets leaked once become forever-usable footholds.
Mental model
Airport security for every box of parts into a jet build
Supply chain hygiene
You don't trust a part because the box 'says Boeing' — you check the manifest, the serial number, and the seal before it enters the hangar. Same for a trendy pip package that vanished: its next release could be a trojan in an ML build.
How it works
- Inventory: SBOM of images, packages, and datasets used by each pipeline.
- Pin: by immutable digest/hash (not floating tags); reproducible installs.
- Sign: artifacts carried with provenance/attestation; merge gates verify signatures.
- Scan: vulnerabilities and malware on images and wheels; acting on findings before merge.
- Least privilege: pipeline credentials scoped and rotated; secrets never in pipeline logs.
Signed supply-chain gates.
In production
- A training cron pins its pytorch+cuda image by digest, scans it in CI, and merges only with a signed SBOM.
- A feature pipeline refuses prunes of floating `latest` tags; every dataset snapshot carries a pinned hash.
- A model registry attests provenance (who built, from what commit) for every registered artifact.
Best practices
- Pin images/wheels/datasets by digest; forbid floating tags in release paths.
- Generate SBOM per build; scan and gate merges on findings.
- Sign artifacts and verify at every consumption point, not just registration.
- Scope pipeline credentials to path/time and rotate; scan CI logs for leaked secrets.
Common mistakes
- `pip install latest` in CI — tomorrow's build pulls a stranger.
- Floating image tags silently rebasing training environments.
- No hash on upstream datasets — 'it was fine last week' becomes the provenance model.
- Over-privileged service accounts with write access to artifacts and registry.
Debugging tips
- A build changed behavior with no code change — diff the dependency/index lock; a floating tag or index shifted.
- Scan suddenly failing on an old image — rotate to a patched digest instead of adding exemptions.
- Signatures fail on merge — attestation mismatch (rebased artifact); re-attest cleanly rather than bypassing.
Interview checkpoint
Quick quiz
Pinning a dependency 'by digest' means…
An SBOM is only useful at build time.