Report Observability: Monitoring, Freshness, Adoption
If nobody can see a failure, nobody can trust a number
Trusted reporting is observable: automation monitors refresh failures, data drift, performance and adoption; alerts ride the pipeline; and dashboards tell stakeholders when the number 'is real'.
What you will learn
- Run freshness, drift and performance checks on the report chain
- Alert on failures with owners and retries
- Track adoption and signal stale data to readers
What is this?
Report observability measures the chain that produces trust: does the model still refresh on time (freshness)? Do the numbers still match the source (drift)? Do reports render within SLA (performance)? Do people actually use them (adoption)? Automation captures failures into alerts with owners — so a silent 3am refresh failure never becomes a wrong board meeting.
Why do you need it?
- Untracked stale data quietly erodes every chart.
- Failures caught early cost nothing; failures at the demo cost everything.
- Adoption data drives honest retirement of dead reports.
- Observability is what separates 'publishing' from 'operating' BI.
Mental model
A smoke detector with the right battery
Report observability
The detector (monitoring) fires when the house starts to smoke — only if it's wired to beep somewhere a human sleeps (alerting) and someone changes its battery (ownership). Observability is the detector + battery + alarm path, installed before the fire.
How it works
- Freshness checks: max(dim_date)/max(refresh) vs SLA, automated.
- Drift checks: model/metric vs control total deltas over thresholds.
- Performance checks: report load and query latency percentiles.
- Adoption: usage telemetry per report (views, users, cadence).
- Alerting: failures route to the owner with retry and on-call escalation.
Observe each link of the trust chain.
Examples
The observable report contract
report: monthly_execowner: pmo@corpchecks: freshness: max(loaded_at) <= 2h, SLA 06:00 drift: reconcile vs control total <= 1% perf: p95 render < 3s adopt: last_viewed rolling 30dalerts: - freshness_fail -> oncall, retry x3, page at 25msignals in app: - "Data as of {timestamp}" on every headerRetirement rule: 60d no views -> reviewChecks + owners + alerts + surface signals — the mechanical parts of a report people can rely on.
In production
- A nightly model refresh failure triggers an on-call alert before the exec deck renders.
- A drift check silently suspects a broken join and freezes the report behind a banner.
- Adoption telemetry retires 12 unread dashboards in the quarterly sweep.
Best practices
- Automate freshness + drift + perf checks; humans only review alerts.
- Alert with owner + escalation, not a log line nobody reads.
- Surface 'data as of' on reports so staleness is visible, not assumed.
- Use adoption data to retire — a catalog audits itself.
Common mistakes
- Monitoring 'the data is there' but not 'the data is fresh/worth it'.
- Alerts with no owner, so failures sleep between shifts.
- Freshness checked but drift never — accurate-looking wrong numbers.
- Ignoring adoption — the catalog rots in plain sight.
Debugging tips
- Executive number wrong → run drift check against control; the answer is usually model drift, not report code.
- Silent weekend refresh failures → alert path missing an owner or on-call.
- Everything 'green' but nothing used → add adoption as a pillar; green doesn't mean valuable.
Interview checkpoint
Hands-on lab
- Compute freshness and drift gates in SQL
- Calculate an adoption signal from usage logs
- Wire the alert conditions logically
Automate Freshness, Drift and Adoption Checks
45 min · advanced
Quick quiz
A report shows on time but the numbers are wrong. Which check catches it?
An adoption metric is part of observability.