Data Observability
From 'is it down' to 'why is this wrong'
Observability goes beyond alerts: queryable metrics/logs/traces of every run and lineage let you answer 'why is it like this' after the fact, turning platforms from monit-orable to debuggable.
What you will learn
- Distinguish monitoring (known conditions) from observability (unknown questions)
- Instrument runs with rich, queryable metadata: lineage, logs, timings
- Use trace-like propagation from source to sink for root-causing
What is this?
Monitoring answers predetermined questions ('is lag > X?'); observability answers questions you didn't think to ask by keeping rich, queryable telemetry: every run's inputs/outputs, lineage edges, timing breakdowns, logs and a propagation identifier (run_id) that stitches the path from a source row to a dashboard number. Data observability is the sum: metrics you can slice, logs you can grep, traces you can follow, and history you can diff.
Why do you need it?
- Alerts only cover the failures you predicted; the interesting incidents are the ones you didn't.
- A 'wrong number in the dashboard' can't be predicted — you need to trace backwards through runs and transforms.
- Run-metadata history lets you diff 'why did this change' between runs instead of guessing.
- Queryability is the multiplier: metrics + logs + lineage in one store costs less than a decade of 'investigate-by-email'.
Mental model
A flight data recorder + ATC replay
Data observability
A recorder doesn't predict turbulence (like alerts), but when something odd happens, it lets you replay the flight: altitudes, headings, comms, in sequence. Observability lets you replay a pipeline: runs, volumes, transforms, timings, in lineage order.
How it works
- Metrics: volume, freshness, lineage counts per table/run stored with dimensions for slicing.
- Logs: structured, centralized logs per job step with levels, trace_id and run_id.
- Traces: propagate a run_id from extraction through transforms to output; each step logs its inputs/outputs and duration.
- History: store run manifests (code version, schema, row counts) so any past run is inspectable.
- Affinity: lineage graph + metrics = answer 'which upstream feeds this, and which one regressed?'.
- UX: a catalog browse — click a table, see its health, its lineage and its last 30 runs.
Telemetry types and what each one answers.
In production
- An analytics team notices the churned-users number changed; using lineage and run manifests they trace it to a changed join in the v2.3 model deployed two hours earlier.
- A streaming org stores per-window metrics plus the code version; a KPI regression is diffed against previous runs to isolate the exact window and version.
- A warehouse probes a mysterious lag by following trace_id from a source Kafka partition through the enrich job to the final table, finding the stray sleep in one transform.
Best practices
- Push a run_id into every step and every output row (as a column) — it's the glue of traces.
- Store at minimum: code version, input/output counts, timing per step, and row-level sample of outputs.
- Make telemetry queryable via the same SQL the team already knows.
- Diff-friendly: keep history cheap so 'compare today to yesterday' is one query.
Common mistakes
- Collecting telemetry into 5 vendor silos that can't be joined.
- Logging unstructured text with no run_id — unelatable magic.
- Recording metrics but no 'code version + inputs', so traces dead-end at 'something changed'.
- Observability used only after incidents; the daily/hourly use is what pays for it.
Debugging tips
- Rule of three: when investigating, pull metric, log and lineage view of the same run simultaneously.
- If two runs differ, diff their manifests (version, volume, timings) — the delta names the culprit.
- Trace from the symptom backward: dashboard → final table run → upstream join → source row sample.
- When nothing shows a difference, compare consumed vs produced volumes per step; a silent drop locates the losing transform.
Interview checkpoint
Hands-on lab
- Carry run_id from load through transforms into outputs
- Reconstruct the trace (lineage + metrics) of one run
- Diff two runs to diagnose 'why is this wrong'
Propagate run_id Across a Transform Chain
30 min · intermediate
Quick quiz
A good proxy for 'this pipeline is observable' is:
A dashboard full of alerts is the same thing as observability.