Dataset Registries & Catalogs
Make every dataset findable, owned and trusted
A data catalog is the searchable front door to your datasets: discoverability, ownership, schema and quality metadata, and the governance that prevents 'which dataset should I use?'
What you will learn
- List what a data catalog records about each dataset
- Use a catalog to govern access and ownership
- Distinguish registry (versions) from catalog (discovery)
What is this?
A dataset catalog is a metadata store that indexes datasets: description, schema, owners, quality scores, freshness and current version. Registries track versions of specific datasets; catalogs make the whole estate discoverable, so an engineer asks the catalog 'which dataset should I use?' instead of asking a colleague.
Why do you need it?
- Dataset sprawl ('us_v2_final_old.xlsx') is unmanageable without a browsable index.
- Ownership metadata answers 'who do I page when this dataset is broken?'
- Quality and freshness scores let consumers judge fitness before they build.
- Catalogs encode governance: who may access, who may republish, what lineage is required.
Mental model
A library catalog vs the stacks
A data catalog vs raw storage
You rarely wander the stacks blind; you search the catalog, check availability and format, then fetch. A data catalog is the card file for datasets — search, metadata, availability, shelf number (storage path).
How it works
- Ingestors register datasets on publish: path, schema, version, owner, description.
- Profilers attach quality metadata: null rates, distinct counts, freshness.
- Search and faceting let users find by name, tag, project or quality.
- Permission systems tie to the catalog: access is granted per dataset in the catalog.
- Lineage from the catalog connects dataset versions to features and models.
Catalog layers over storage and lineage.
In production
- An engineer searches 'churn features' and finds three candidates with freshness scores and owners.
- A data team sets ownership on every dataset; the catalog flags unowned datasets quarterly.
- Regulatory review scrolls the catalog's 'datasets feeding model X' lineage in two clicks.
Best practices
- Register every published dataset — a dataset absent from the catalog is a liability.
- Require owner, description and SLA (freshness) at registration.
- Auto-profile quality so metadata doesn't rot after onboarding.
- Tie access permissions to the catalog, not to ad-hoc file shares.
Common mistakes
- Catalog becomes a 'nice-to-have' wiki that falls behind reality.
- Versioning in filenames while the catalog records no version linkage.
- No ownership field — the classic 'nobody owns this dataset' black hole.
- Quality metadata exists but is never recomputed and goes stale.
Debugging tips
- 'Wrong data in production' — check the catalog's lineage for the dataset and its latest version.
- Can't find a dataset? Search by owner, tag and project; if missing, registration was skipped.
- Stale quality scores? Re-run profilers on schedule so the catalog reflects reality.
Interview checkpoint
Hands-on lab
- Model dataset metadata as records in a versioned store
- Register datasets, set owners and quality scores
- Query the catalog for discoverability and governance checks
Build a Minimal Dataset Catalog
25 min · intermediate
Quick quiz
Which metadata is essential to keep in a data catalog for every dataset?
A data catalog that falls behind reality is worse than no catalog.