Streaming
Intermediate 15 min read Streaming

Kafka Connect

Streams from systems you don't want to code

Kafka Connect is a framework of reusable connectors that move data between Kafka and external systems — databases into Kafka (sources), Kafka into sinks (S3, Elasticsearch, warehouses) — with exactly-once and schema management handled by the platform.

kafka connectconnectorsourcesinkCDCDebeziumschemaexactly-once
Prerequisites: Kafka Topics and Partitions
Before we start

What you will learn

  • Explain the source/sink connector model and configuration
  • Describe how Debezium turns DB changes into events
  • Select connectors for lake, warehouse and search sync
Introduction

What is this?

Kafka Connect is a distributed integration framework: connectors defined declaratively move data into Kafka from sources (databases via CDC, files, SaaS) and out to sinks (object storage, Elasticsearch, warehouses, JDBC). It runs as workers on a cluster, hands offsets and schema over to Kafka's exactly-once semantics, and makes new integrations a YAML/JSON change instead of a new service.

Motivation

Why do you need it?

  • Standardizing on one integration layer avoids N bespoke ingestion services.
  • Connectors handle the hard bits: batching, retries, schema, offset tracking (change data capture).
  • CDC connectors (Debezium) stream row-level changes with zero writes to the source.
  • Managed connectors (Confluent Cloud, connectors on CDP) take ops off your plate.
Think about it

Mental model

Imagine

A universal adapter for every appliance

Think of

Kafka Connect connectors

Instead of hiring a new electrician for each brand of appliance, you use a standard socket and the right plug (connector) — plug it in, configure it, and the power flows both ways (source or sink).

Mechanics

How it works

  • A connector definition declares a source or sink, its topic, and its options (schema, partitioning).
  • Source connectors poll external systems, produce records to topics, and track how far they've read.
  • Sink connectors consume a topic and write to an external target with the right format.
  • Workers distribute connector tasks; fixing one connector doesn't touch others.
  • Change data capture sources (Debezium) tail the database's binlog/WAL producing insert/update/delete events.

Kafka Connect source and sink paths.

Source systems
Source connector
Kafka topics
Sink connector
Targets
Real world

In production

  • Debezium streams every INSERT/UPDATE/DELETE in PostgreSQL into Kafka, feeding a search index and a warehouse.
  • A file-source connector watches an FTP drop and publishes each file's events, retrying on partial reads.
  • An S3 sink connector lands topic data as Parquet partitioned by day, ready for Athena queries.
Guidelines

Best practices

  • Give each connector its own topics and schemas (via schema registry) so consumers evolve safely.
  • Set connector tasks to match partition count and throughput, not arbitrary values.
  • Use exactly-once (EOS) mode on the connector where sinks support transactions.
  • Monitor connector offsets and lag per connector, not just consumers.
Watch out

Common mistakes

  • Running every connector on a single worker with no fault tolerance.
  • Trying to reshape data inside connectors — do the heavy lifting in stream processing, not adapters.
  • Ignoring Debezium's schema changes (ALTER TABLE) breaking downstream schemas.
  • Consumers of CDC events treating only 'insert' — delete events must be handled too.
Root cause analysis

Common issues

Error
Connector fails when the target schema changes
Root cause

Schema drift not mirrored in the connector config or registry

Fix

Evolve the schema via registry compatibility checks, and restart the task after.

Error
CDC lag builds as table volume grows
Root cause

Too few source tasks or per-partition batching

Fix

Increase tasks to match partitions and lower batch sizes; add indexes on source.

Error
Sink writes partial/duplicate files on restart
Root cause

Non-exactly-once connector mode

Fix

Enable exactly-once/EOS on the connector and use idempotent targets.

When things break

Debugging tips

  • Connector offsets (source position) are where replay lives — resetting them re-syncs a system.
  • Sink errors usually show in task status logs; check both task state and target partially-written files.
  • CDC missing rows → verify the connector reads with the right snapshot.locking.mode.
  • Validate connector schema evolution against the registry before deploys.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which connector scenario uses Debezium-style CDC?

2

Offsets for source connectors are what make re-sync/replay possible.

Go deeper

Official documentation

Kafka Connect docs
docs
Debezium connector docs
docs