Observability Fundamentals

• You cannot fix what you cannot see — most production incidents are prolonged by gaps in the telemetry, not by lack of effort.
• Monitoring tells you a service is down; observability tells you which dependency, which code path and which deploy caused it.
• The three pillars answer different questions: metrics tell you what is happening, logs tell you what happened, and traces tell you where it happened in the call graph.
• White-box versus black-box thinking changes what you collect: internal instrumentation versus synthetic checks from outside.
• Cardinality decisions made on day one are painful to reverse later, so understanding series growth is a core skill.

The dashboard gauges are metrics — speed, fuel, temperature. The driver's memory of recent events is logs. A GPS route replay that reconstructs every turn and stop is a trace. You can drive with only a speedometer, but you cannot explain a near-miss or reroute efficiently without the full picture.

• Instrumentation inside the application emits telemetry as things happen: metrics are recorded on counters and gauges, logs are written per event, and spans are created per unit of work.
• Exporters or agents ship that telemetry to collectors and storage backends such as Prometheus, Elasticsearch or Jaeger.
• Consumers query the stored data: dashboards render it for humans, alert rules evaluate it automatically, and ad-hoc query languages let engineers explore.
• A trace ID and shared labels (service, environment, region) let a single user request be correlated across every service it touches.
• Engineers drill from a suspicious dashboard spike down into the relevant trace, then into the specific log lines, to root-cause in minutes instead of hours.

• Every service exposes /healthz and /metrics from day one, so rollouts are gated on real signal instead of guesswork.
• On-call engineers start every incident on a golden-signals dashboard — latency, traffic, errors and saturation — before touching logs.
• Platform teams publish a 'measure everything by default' standard so new services emit telemetry without special effort.
• Black-box synthetic probes complement white-box metrics, catching outages that internal instrumentation never reports because the app is dead.
• Telemetry volume and cardinality are reviewed quarterly so cost and storage stay predictable as services grow.

• Instrument from day one; retrofitting telemetry into a running service is slow and error-prone.
• Standardize naming and structure across services, e.g. http_requests_total{method,status,route} everywhere.
• Review cardinality in code review: a new label with unbounded values multiplies the series count.
• Tag every signal with stable identifiers: service name, version, environment and region.
• Default every service view to the golden signals so the most important questions are answered first.

• Treating dashboards as observability — a dashboard is only as good as the data feeding it, and you can only monitor what you instrument.
• Confusing monitoring with observability: monitoring checks known conditions, observability lets you investigate unknowns.
• Adding high-cardinality labels such as request IDs or user IDs to metrics, exploding the number of stored series.
• Storing logs as free-form text, which makes queries and cross-service correlation guesswork.
• Collecting telemetry only in production, so staging bugs ship because they were never visible.

• Drill top-down: golden-signals dashboard, then trace, then specific log lines — never start with raw log dumps.
• Reproduce an issue in staging with the same instrumentation enabled so you can compare signals.
• Take a slow response's trace ID and follow the call graph; the widest span gap points at the bottleneck.
• Compare telemetry before and after the last deploy to isolate a regression to a specific change.
• Verify end-to-end: if a metric is missing, check the exporter, the scrape network path and retention before suspecting the app.