Schema Evolution
Change the shape without breaking the readers
Schemas change. Handling that deliberately — additive-first, typed contracts, registry/versioning, consumer coordination — keeps producers fast and consumers unbroken.
What you will learn
- Apply additive-first evolution rules
- Distinguish backward/forward compatibility
- Use registries and contracts to coordinate producers and consumers
What is this?
Schema evolution is the discipline of changing data shape without breaking readers: add fields, never rename/retype in place; keep old versions readable (backward compatible) and new writers acceptable to old readers (forward compatible); and use a registry — Avro/Protobuf schema registry or dbt-style contracts — so compatibility is checked, not assumed.
Why do you need it?
- Schema changes are the quiet way pipelines break at 3am.
- The registry turns 'will it break?' into a compatibility check at schema-commit time.
- It is the same idea in CDC, event streams, and warehouse marts.
Mental model
A form with a 'name' field that must never silently become 'full_name'
Schema stability
Readers wrote to 'name'; if you rename it, they stop reading. Adding an optional 'middle_name' is safe; renaming or retyping is how old forms misread new data.
How it works
- Additive first: new optional fields pass old and new readers; nothing existing changes meaning.
- Backward compatible: old readers can still consume new data (added fields defaulted).
- Forward compatible: new readers handle old data (new fields are optional/nullable).
- Registries check each schema against the last N compatible versions at registration time.
- Contracts (dbt) fix expected shape/output tests; warehouse hits them before dashboards do.
Production- and consumer-safe evolution.
Examples
Compatibility by adding, not mutating
-- v1: {"order_id": 1, "amount": 25.0}-- v2 proposal: ADD field (compatible):{"order_id": 1, "amount": 25.0, "currency": "USD"} -- v3 proposal: RENAME field (breaking): {"order_id": 1, "total": 25.0} # renames amount -> total -- Choose v2. The registry rejects v3 automatically via compatibility checks.Additive evolution keeps both old and new readers correct; rename/retype is where consumer breakage hides.
In production
- A streaming team registers Avro schemas so an added event field doesn't kill consumers.
- A dbt contract asserts warehouse mart columns (nullable, type) in CI, blocking breaking model changes.
- A lake landing validates a JSON schema on write so downstream reads never see a silent rename.
Best practices
- Make most changes additive; plan renames as 'add new + deprecate old' over a window.
- Keep optional new fields nullable/defaulted so old readers survive.
- Register versions in a registry and enforce compatibility in CI.
- Communicate deprecation windows; silent data changes are the real crime.
Common mistakes
- Renaming a column 'for clarity' and breaking three dashboards the same night.
- Retyping an ID (int->string) so joins silently stop matching.
- Dropping a field before consumers have migrated; the data dies before the code migrates.
Debugging tips
- A 'column not found' breakout after a deploy = a breaking schema change shipped; roll back or add migration.
- When joins match nothing, diff the key types across the two schemas first.
- Registry push failures are the cheapest possible error — the check you want to see.
Interview checkpoint
Quick quiz
Adding an optional nullable field is backward compatible for almost all readers.
A registry's job is mainly to…