Dataset Versioning
Intermediate 14 min read Dataset Versioning

Hashing and Lineage

Prove what the bytes are, and where they came from

Content hashes make datasets machine-verifiable; lineage records the family tree — source, transformations, and what consumed it. Together they make data auditable end to end.

content addressinghashlineageprovenancedata forestdata versioning
Prerequisites: Why Version Datasets
Before we start

What you will learn

  • Compute content hashes that uniquely identify a dataset
  • Explain how lineage graphs connect sources to models
  • Use hashes to prove a dataset is unchanged
Introduction

What is this?

Content addressing hashes a dataset's bytes so the identifier IS the content — a changed file gets a different hash. Lineage is the graph of how a dataset came to be: raw source → transforms → feature table → training data → model. Hashes verify; lineage explains.

Motivation

Why do you need it?

  • File names lie; content hashes don't — 'clean_data.csv' can change names and contents independently.
  • Hashes make versions diffable at scale (compare datasets without comparing every row).
  • Lineage answers 'where does this number come from' for auditors and debugging.
  • Hashes + lineage let you build trust automatically: verify inputs match before promoting a model.
Think about it

Mental model

Imagine

Fingerprints and a family tree

Think of

Hashing and lineage

A fingerprint (hash) uniquely identifies one person. A family tree (lineage) shows how they're related to everyone else. You need both: identity and history.

Mechanics

How it works

  • A hash function (e.g. SHA-256) is computed over the dataset's bytes or rows, producing a stable identifier.
  • Storing files by hash (content-addressed storage) makes duplicates detectable and immutability natural.
  • Each transformation records inputs → output, creating directed edges.
  • Lineage tools (DVC, data catalogs, MLflow) walk these edges to answer 'what fed this model'.
  • Hashed references in configs let pipelines verify inputs before running.

Content addressing plus a lineage graph.

Raw source
Hash
Transform
Output hash
Model
Hands-on

Examples

Hash, diff, verify

Hash, diff, verify
import hashlib
def hash_file(path, chunk=2**16):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for block in iter(lambda: f.read(chunk), b""):
            h.update(block)
    return h.hexdigest()
 
# pin the dataset in the training config
config = {"data": {"version": hash_file("dataset.parquet")}}

The config stores the content hash rather than a filename. If the file changes, the config no longer matches and the pipeline can refuse to run silently on the wrong data.

Real world

In production

  • A pipeline asserts the upstream dataset hash before training, blocking silent corruption.
  • A catalog stores hashes per version, so the platform can detect unauthorized rewrites.
  • An on-call engineer traces a metric change to a transform commit by walking lineage edges.
Guidelines

Best practices

  • Use content hashes, not timestamps, as the identity of immutable datasets.
  • Record both input and output hashes at every transform boundary.
  • Store lineage as machine-readable metadata, not prose in a README.
  • Automatically verify hashes before any training or promotion step reads the data.
Watch out

Common mistakes

  • Hashing a filename instead of its bytes.
  • Recording lineage in a wiki that nobody updates.
  • Reordering rows and assuming the same content — sort before hashing or hash sorted output.
  • Ignoring column-order/schema changes when hashing frames.
When things break

Debugging tips

  • Hash changed but 'nothing changed'? Include schema and sorted row order in the hashing contract.
  • A model score moved without code changes? Diff upstream data hashes through the lineage.
  • Hashed files unreachable? Content-addressed stores fail loudly when a hash has no object.
Interviews

Interview checkpoint

Practice

Hands-on lab

Hash, Diff and Verify a Dataset
20 min · intermediate
In progress
Objectives
  • Compute content hashes for dataset files
  • Prove a renamed-but-identical file hashes the same
  • Prove a single changed row changes the hash

Hash, Diff and Verify a Dataset

20 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which property best describes a content hash?

2

Lineage only matters for file storage and not for model trust.

Go deeper

Official documentation

DVC — metadata & lineage
docs