Data Lakes
Intermediate 11 min read Data Lakes

Schema-on-Read vs Schema-on-Write

Flexibility with guardrails is the mature answer

Schema-on-read keeps data flexible and fast to ingest; schema-on-write enforces correctness at the boundary. Modern lakes pair them: validate on write for trusted zones, explore on read for discovery.

schema-on-readschema-on-writeevolutionvalidationdiscovery
Prerequisites: Lake Fundamentals
Before we start

What you will learn

  • Compare the two schema strategies with costs and benefits
  • Decide where each is safest
  • Design evolution rules that don't break consumers
Introduction

What is this?

Schema-on-read applies structure when data is queried; schema-on-write validates and types data as it lands. A mature lake does both: schema-on-read for discovery and early-stage experiments, schema-on-write (with tolerated evolution rules) for zones feeding dashboards and ML.

Motivation

Why do you need it?

  • The 'cheap ingestion' of schema-on-read becomes expensive when 400 dashboards re-discover the shape.
  • Enforcing early prevents the most common silent data bugs: type drift and missing fields.
  • The right mix is the difference between agility and a swamp.
Think about it

Mental model

Imagine

Reading a recipe written in someone else's shorthand

Think of

Schema-on-read

You can cook it eventually, but every cook interprets the shorthand differently unless a standard (schema-on-write) was set when the card was filed. Discovery loves flexibility; production loves checked recipes.

Mechanics

How it works

  • Schema-on-read: engine infers schema from the file at query time; change the file, change the meaning.
  • Schema-on-write: a validator (Spark schema, dbt contract, JSON schema) rejects or casts on ingest.
  • Evolution rules: additive (add column) safe; breaking (rename/ret-type) needs consumer coordination.
  • Zone policy: raw allows anything, curated enforces typed schema and quality gates.
  • Tooling: kafka schema registry, formal validation in dbt models, Iceberg/Delta still rely on your gates.

Validation intensity across zones.

Raw
Curated
Analytics
Dashboards / ML
Hands-on

Examples

Schema evolution done deliberately

Schema evolution done deliberately
-- Legacy pipe (schema-on-read): a new field silently appears.
SELECT payload.amount FROM read_json('lake/raw/events/*.json');
 
-- Trusted pipe (schema-on-write): keep the raw JSON, but guarantee a typed view.
CREATE OR REPLACE TABLE curated.orders (
  order_id BIGINT NOT NULL,
  amount DECIMAL(12,2) NOT NULL,      -- rejected if absent
  created_at TIMESTAMP
);

Raw preserves fidelity; curated enforces what dashboards may rely on.

Real world

In production

  • An ingest service accepts raw JSON unchanged, then a regular housekeeping job confirms a stable typed schema for the daily models.
  • A dbt model defines a CONSTRAINT test so 'schema changed' fails CI instead of the morning dashboard.
  • A schema registry rejects an Avro event missing a required field at the producer boundary.
Guidelines

Best practices

  • Schema-on-write for anything a dashboard, alert, or ML feature reads.
  • Make evolution additive by default: new optional columns, never rename/retype in place.
  • Version the schema (registry/contract) and deprecate old versions on a timeline.
  • Keep raw fidelity (schema-on-read) where consumers consciously own the risk.
Watch out

Common mistakes

  • Putting every discovery need on schema-on-write and freezing the raw zone's agility.
  • Or the opposite: no validation at all, and every dashboard rebuilds the world from guesses.
  • Renaming a column in the writer without coordinating with the 3 downstream catalogs.
When things break

Debugging tips

  • When a dashboard breaks 'out of nowhere', diff the last deployed schema/contract against the model's expected shape.
  • A unit of 'schema drift' is usually a defaulted/new optional field flowing through an unrecorded producer change.
  • Check registry evolution rules: a 'backward incompatible' flag surfaces early, a silent one later.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

A curated analytics zone typically deserves stricter schema enforcement than the raw zone.

2

The safest default schema change is…

Go deeper

Official documentation

Avro schema evolution types
docs
dbt contracts
docs