Data & Dataset Versioning
Tying behavior to its inputs
Corpora, prompts, embeddings and eval sets change over time. Versioning datasets links a model's behavior to the exact data it was built from — essential for reproducible evals and auditable AI.
What you will learn
- Version datasets with hashes, manifests and lineage
- Reproduce an eval or training run from a data version
- Handle drift between corpus versions cleanly
What is this?
Dataset versioning gives each corpus, eval set or prompt-embedding snapshot an identity: a content hash plus a manifest of provenance. It lets you answer 'which data produced this model/score?' and roll a system back when a data change backfires. Without it, 'the eval got better' is unverifiable.
Why do you need it?
- A changed corpus silently improving/staining an eval makes decisions impossible.
- Support escalations need to reproduce the exact dataset a behavior came from.
- ML compliance and audit require knowing which data shipped with which model.
- Rolling back is trivial when you can reinstall a data version, impossible when you can't.
Mental model
A recipe book with revision numbers
Dataset versioning
The finished dish depends on the exact revision of the recipe. Split pea soup v3 is not v2 with a garnish — someone has to know what actually went in. Data manifests are the revision numbers of your content.
How it works
- Content is hashed; the dataset identity is the manifest or the root hash.
- DVC (or a hash-based store) tracks metadata and versions files in object storage.
- Pipeline steps record which input versions produced each output version (lineage).
- Eval jobs pin dataset + prompt + model + dependency versions in one run record.
- Rollback = re-copying a prior manifest's content and re-running dependent stages.
Lineage from corpus to eval score.
In production
- An eval harness records corpus version, prompt version, model and SDK version in every run so any score is replayable.
- A RAG pipeline debuts a new corpus version behind a feature flag, diffing retrieval quality before promoting.
- A compliance team pulls the exact eval snapshot that a released bot passed, from a manifest alone.
Best practices
- Hash everything content-derived; never rely on timestamps alone.
- Record lineage at every stage — inputs in, outputs out, with versions.
- Pin corpus, prompts, model and dependencies together in run records.
- Promote dataset changes through the same review you give code changes.
Common mistakes
- Versioning by filename/date, which duplicates silently (v2_is_FINAL.json).
- Letting eval jobs float on 'latest' data so scores drift between runs.
- Not versioning prompts as data — prompt updates are the sneakiest dataset changes.
- Deleting old corpus versions 'to save space' and losing the ability to rollback.
Common issues
Eval improved but nothing in code changedEval set or corpus version floated to a newer snapshot
Pin eval data and corpus versions; diff manifests on any unexplained movement.
Can't reproduce a model's eval scoreNo run record tying data, prompts and model together
Add a run manifest capturing every pinned input version.
Rollback attempt resurrects corrupted chunksCorpus version deleted; rollback re-uses stale cache
Retain versioned snapshots for the rollback window and clear derived caches on cutover.
Debugging tips
- When an eval flips unexpectedly, diff the corpus manifests between runs first.
- If retrieval behavior changed with no code change, the corpus version drifted — check the pinned manifest.
- Trace a strange score to lineage: which data version, which prompt version, which model.
Interview checkpoint
Hands-on lab
- Compute a content-hash manifest for a corpus
- Pin corpus + prompt + model ids into a run record
- Roll back to a prior manifest and confirm reproducibility
Manifest-Pin a Mini Corpus
25 min · intermediate
Quick quiz
What is the primary identity of a dataset version?
Prompts are data when it comes to reproducibility.