Streaming
Intermediate 15 min read Streaming

Schemas and Schema Registry

Versioning the contract between producer and consumer

Streams are only as good as their schema. A schema registry versions the messages on each topic so producers and consumers evolve safely — and it validates events before they're consumed.

schemaschema registryavroprotobufbackward compatibilityevolution
Prerequisites: Kafka Topics and Partitions
Before we start

What you will learn

  • Explain why streams need a schema contract
  • Use registry compatibility modes for safe evolution
  • Choose Avro/Protobuf/JSON-Schema for a topic
Introduction

What is this?

A schema registry is a versioned catalog of the message formats for every topic in a Kafka infrastructure. Producers register schemas (Avro, Protobuf, JSON-Schema) and get an ID; payloads carry the ID and consumers deserialize against the registered version. Compatibility modes (backward, forward, full) enforce what schema changes are allowed, so old consumers never break on new producers.

Motivation

Why do you need it?

  • Without schemas, a field rename or type swap silently corrupts every downstream consumer.
  • The registry makes schema a deployable, reviewable contract — not an implicit assumption.
  • Compatibility checks catch breaking changes at registration time, before they burn a fleet.
  • Compact wire formats (Avro) also save bandwidth and broker memory.
Think about it

Mental model

Imagine

A standards board for forms

Think of

Schema registry

Every form created must be approved and versioned by the board (registry), and each printed copy carries its version number. When the paper size changes, the board accepts a backward-compatible revision, and ancient printers can still read it or clearly error.

Mechanics

How it works

  • Producers register a schema; the registry assigns a global schema ID and enforces compatibility.
  • Serialized messages include the schema ID (usually a magic byte + 4-byte ID).
  • Consumers fetch the schema for that ID and deserialize — even the first event they've ever seen.
  • A new producer version registers as schema v2 subject to the topic's compatibility rule.
  • Compatibility modes: BACKWARD (new schema reads old data), FORWARD (old schema reads new), FULL (both), NONE.

Schema registry in the produce/consume loop.

Producer
Registry
Topic records
Consumer
Contract
Real world

In production

  • A team adds a 'country' field to an event via schema v2 — FULL compatibility approves, and old consumers ignore the new field.
  • A breaking rename fails registration under BACKWARD, blocking a deploy instead of a data incident.
  • Avro payloads shrink JSON logs 60%, lowering broker network I/O.
Guidelines

Best practices

  • Default to FULL or BACKWARD compatibility on central topics.
  • Version record schema AND key schema separately (they're different contracts).
  • Explicitly decide default vs backward: evolution should be explicit, not accidental.
  • Use IDs at runtime, never schema text — the registry is the single source of truth.
Watch out

Common mistakes

  • Registering on NONE and breaking consumers silently.
  • Adding a required field without a default under BACKWARD (breaks old consumers).
  • Renaming a field 'by refactoring' instead of aliasing under compatibility rules.
  • Forgetting the key schema — key evolution changes partitioning semantics.
Root cause analysis

Common issues

Error
Consumers fail to read new producers after schema change
Root cause

Registered with NONE and produced incompatible data

Fix

Evolve schemas under FULL/BACKWARD compat; fix consumers or explicitly re-publish with versioning.

Error
Schema registry single node is a SPOF
Root cause

Registry runs standalone

Fix

Run the registry in a multi-node/HA mode backed by Kafka.

Error
Record uses stale schema ID
Root cause

Producer cached an ancient registered ID

Fix

Clear producer schema cache and republish; use compatibility-safe forward refs.

When things break

Debugging tips

  • DeserializationException on a consumer → the schema for the message ID has changed incompatibly.
  • Registration rejected → the compatibility rule failed; read the diff before forcing NONE.
  • Record size balloon → check you're using compact encoding and not stringified JSON.
  • Which schema is deployed? Registers on the registry UI, not the repo.
Interviews

Interview checkpoint

Practice

Hands-on lab

Evolve Avro Schemas Safely
30 min · intermediate
In progress
Objectives
  • Define an Avro record and serialize/deserialize with fastavro
  • Prove a backward-compatible evolution (adding an optional field) still deserializes old payloads
  • Demonstrate a breaking change and why registry compatibility checks reject it

Evolve Avro Schemas Safely

30 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Which change breaks BACKWARD compatibility?

2

Schema evolution should be enabled explicitly and reverted only with discipline.

Go deeper

Official documentation

Confluent Schema Registry
docs
Avro spec
docs