Staging, Serving and Data Vault Layers
Bronze, silver, gold — and why the raw layer is sacred
Warehouses are layered: raw/staging, cleansed, modeled serving and sometimes data-vault-style staging for auditability. Layers let teams separate concerns and keep raw truth immutable.
What you will learn
- Describe the bronze/silver/gold data flow
- Explain why raw must stay immutable
- Compare medallion layering with data-vault staging
What is this?
Warehouse layering separates transformation stages: bronze (raw snapshots, immutable), silver (cleansed, deduplicated, entity-aligned) and gold (modeled serving for BI). Data-vault adds auditable staging (hubs/links/satellites) for high-uncertainty or regulated sources. Layers make pipelines debuggable, backfillable and traceable.
Why do you need it?
- Immutable raw is the forensic record — any modeled table can be rebuilt from it.
- Layer separation gives each stage its own failure mode, owner and SLA.
- Governance can differ by layer: raw is locked, gold is curated and certified.
- When a 'silver' bug is found, you repair gold by building from raw, not by patching guesses.
Mental model
Photographic negatives, proofs and prints
Warehouse layers
Negatives are the untouched original (bronze), proofs are cleaned-up drafts (silver), and the final print arrives at the gallery (gold). If a print is bad, you never retouch a copy of a copy — you go back to the negative.
How it works
- Bronze/raw: source snapshots and events land verbatim, timestamped, write-once.
- Silver/curated: dedupe, typecast, conform entities (one customer id), enforce quality tests.
- Gold/serving: fact/dimension models, semantic views and certified aggregates for BI.
- Data vault layers stage hubs (entities), links (relationships) and satellites (attributes) for auditable lineage.
- Each layer has owned refresh jobs; lineage records the path from gold back to bronze.
The layer pipeline and its trusts.
In production
- Every source lands in bronze as a full snapshot; silver builds one clean customer per source; gold models orders, revenue and churn for dashboards.
- A regulated bank keeps a data-vault style audit trail—every change to a customer attribute is a satellite row with valid_from/valid_to.
- When a gold aggregate is wrong, the team backfills from silver in minutes, then diff-reconciles.
Best practices
- Treat bronze as append-only; never let a transform clobber a snapshot.
- Put quality gates between layers: silver should pass expectations before gold builds.
- Give each layer an owner and prefix (bronze_*, gold_*) so lineage is self-describing.
- Document the backfill procedure per layer — restore is a feature, not an afterthought.
Common mistakes
- Letting reports read silver or bronze 'because it's faster' — definitions fork.
- Updating raw in place when a source re-emits, corrupting forensic history.
- Skipping silver and modeling from bronze repeatedly, spreading messy logic.
- Building four layers because the vendor sells four badges, not because you need them.
Common issues
Two gold reports disagreeOne report bypassed gold and read silver
Enforce gold-only access for consumers; keep silver internal.
Backfill can't reproduce a tableRaw was overwritten instead of appended
Make bronze append-only with snapshot timestamps; rebuild from history.
Debugging tips
- Track the layer a discrepancy entered: diff bronze→silver→gold counts to localize.
- If gold is right but a silver table diverges, check the silver quality gates.
- Lineage from gold to bronze tells you which raw wrote the bad value.
Interview checkpoint
Hands-on lab
- Land raw snapshots into bronze
- Curate entities into silver
- Build a gold serving model and backfill it
Layered Warehouse in SQLite
40 min · intermediate
Quick quiz
Which layer should be strictly immutable?
Silver is an optional shortcut that reports can safely bypass.