Production Data Engineering
Beginner 14 min read Production Data Engineering

Monitoring Data Pipelines

From process-is-alive to data-progresses

Real monitoring is measuring what a pipeline actually did — runs, volumes, freshness, lag — and alerting on deviation, not just 'did the JVM exit 0'.

monitoringmetricsrunsvolumelagsynthetic check
Before we start

What you will learn

  • Layer monitoring: infra → job → data → business outcome
  • Instrument the metrics that detect silent failure (volume, freshness, lag)
  • Route alerts by severity and ownership
Introduction

What is this?

Pipeline monitoring measures, over time, whether jobs run, and more importantly whether the data they produce advances as expected: run success/failure, schedule delay, row volumes, freshness of newest rows, consumer lag and external dependency health. The step from 'the process is alive' to 'the data is progressing' separates monitoring that catches real incidents from monitoring that only pages on crashes.

Motivation

Why do you need it?

  • Most data platform incidents are silent: the job exits 0 but loaded yesterday's data, or a source stopped emitting.
  • Volume/freshness/lag metrics detect what process-monitoring can't — the emptiness is the outage.
  • Trends (is volume declining?) catch slow decay weeks before a hard threshold trips.
  • A small set of powerful metrics beats 500 dashboards nobody reads.
Think about it

Mental model

Imagine

A patient with a heart monitor

Think of

Pipeline monitoring

You don't monitor only 'is the patient breathing?' (process alive) — you watch oxygen, heart rate and trends over time. Pipeline monitoring watches the equivalent vitals: volume, freshness, lag, dependency position.

Mechanics

How it works

  • Infra layer: host, cluster, queue metrics (CPU, memory, Kafka lag, executor failures).
  • Job layer: run status, duration, retries, log-level signals from every scheduled/DAG step.
  • Data layer: row counts, freshness, uniqueness, sums — compared against expectation per run.
  • Dependency layer: upstream source health (API reachable, feed landed on time).
  • Business layer: KPI parity (dashboard metric matches a canonical source) — the 'is it right' signal.
  • Each layer emits to a metrics store; alerting thresholds are set from observed baselines.

Four monitoring layers from infra to data advancement.

Infra
Job
Data
Business
Real world

In production

  • An e-commerce platform alerts when the orders landing table's row count drops 40% below its 30-day average — catching a source connector outage within the hour.
  • A media company monitors freshness per table as 'newest event age' and pages the owning squad when a core table exceeds 2h old.
  • A financial platform tracks consumer-group lag and declares an incident when lag exceeds the SLA window, not when a broker crashes.
Guidelines

Best practices

  • Monitor data progression, not just process liveness — empty runs must look different from full ones.
  • Base thresholds on observed baselines with seasonal awareness (weekday vs weekend volumes).
  • Route by ownership and severity: silent-but-important gets a page; noise gets a dashboard.
  • Keep a runbook per alert so the page immediately gives next steps.
Watch out

Common mistakes

  • A wall of dashboards nobody reads and thresholds nobody reviews.
  • Alerting on every anomaly including expected weekdays-off noise — the team mutes everything.
  • Monitoring only the final table, so mid-stream stalls go unnoticed until the end.
  • Missing the 'source didn't produce' case because you only watch your own job status.
When things break

Debugging tips

  • When a metric looks wrong, verify the metric itself first: is the counter wired to the right partition/table?
  • Read trend, not snapshot: one red point is noise; a monotonic decline is an incident.
  • Correlate the failing metric against deploy/version timestamps of pipelines and sources.
  • Reproduce narrowing: from infra→job→data, find the first layer showing deviation.
Interviews

Interview checkpoint

Practice

Hands-on lab

Instrument a Batch with Volume and Delay Metrics
30 min · beginner
In progress
Objectives
  • Track run duration, row volume and schedule delay for a batch
  • Compute volume-vs-baseline deviation and flag outliers
  • Produce a run summary that reads like a monitor's heartbeat

Instrument a Batch with Volume and Delay Metrics

30 min · beginner

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

A batch 'succeeds' but loads zero rows. What catches it?

2

Monitoring a trend is more useful than reacting to a single snapshot.

Go deeper

Official documentation

Metrics and monitoring (Prometheus)
docs
The Four Golden Signals
docs