Storage and Data Costs
The bill that compounds while you sleep
Datasets, checkpoints, artifacts, and training logs quietly outspend compute over time. Control storage with lifecycle tiers, checkpoint retention policies, dedupe and versioning discipline, and budgeted data growth.
What you will learn
- Account for storage across datasets, checkpoints, artifacts, logs
- Apply lifecycle and retention policies
- Control data growth with versioning and tiering discipline
What is this?
Storage and data cost management tracks the compounding estate — source datasets, feature repos, checkpoints, model artifacts, logs — applying lifecycle tiers (hot/cold/archive), retention limits, deduplication, and versioning discipline so the storage bill stays proportional to value, not to accumulation.
Why do you need it?
- Every training run grows checkpoint and artifact inventory structurally.
- Unbounded retention (keep-everything-forever) is the default that saves nothing.
- Data estimates for ML (multi-GB/TB) are optimized for reuse but priced for eternity.
- Versioned stores can dedupe and tier; unmanaged stores stack full-price copies.
Mental model
A warehouse keeping every box forever with no purge
Unmanaged ML storage
Every run is a new shelf (checkpoint), every experiment a crate (artifact), every retrain a carton (dataset snapshot). Except you keep them all, climate-controlled (hot tier), forever. Meanwhile the store runs out of room and pays quadruple — the shelving, not the stock, is the real cost.
How it works
- Inventory: classify storage by kind (dataset, checkpoint, artifact, feature, log) and access heat.
- Lifecycle: hot for active, cold for cold reads, archive for compliance-literacy.
- Retention policies per kind: keep N latest checkpoints, prune failed runs, expire logs.
- Dedupe and tier: versioned stores keep diffs; dedupe identical blobs; encode not replicate waste.
- Govern: budget storage growth; auto-purge by policy with an exception list, not by panic.
Storage lifecycle pipeline.
In production
- A training team keeps the 10 latest checkpoints, purges run waste after a week, and archives final models' data to cold tier — bill drops 60%.
- A dataset repo dedupes across experiment variants; 'copy dataset' becomes a version pointer, not a 4TB copy.
- A log store expires debug logs at 30 days, keeps audit logs for compliance, in archive tier.
Best practices
- Classify storage by kind and heat before setting policy.
- Retain-by-policy, prune-by-authority (auto) with a short exception list.
- Version data by pointer/diff, not by full copy, wherever the store supports it.
- Set growth budgets; treat 'no one can delete anything' as the cost bug it is.
Common mistakes
- Keep-every-forever hot storage 'in case' someone needs it — the entire line item.
- Copying datasets per experiment as a reflex, then counting bytes you accidentally doubled.
- Checkpointing every step on every run, forever, at full price.
- Tiering only the huge object, leaving a million tiny logs full-price.
Debugging tips
- Storage spike with no new data — check failed-run artifacts and log replication gone wild.
- Archive read trips cost more than it saved — you archived a hot-active object; re-tier by actual access.
- Dedupe not working — your version IDs aren't content-addressed; pointer-by-hash fixes it.
Interview checkpoint
Quick quiz
Retrieve-archived-too-early objects is a sign you…
Content-addressed dataset pointers can stop experiments from inventing full 4TB duplicates.