Observability
Intermediate 13 min read Observability

Logging & Structured Logs

Write the truth as structured events

Logs are the event record; structured logs (JSON) are searchable, alertable, and correlate with traces. The craft: request/context details, small logging budgets, sampling, and architecture from app → collector → store → SIEM.

structured loggingJSONlog levelsamplingLokiCloudWatchFlushback
Prerequisites: Observability Foundations
Before we start

What you will learn

  • Emit structured JSON logs with correlation ids
  • Manage verbosity, sampling and retention
  • Pipeline logs agent → store → alert/SIEM
In 60 seconds

Structured logs = one JSON object per event with level, ts, service, trace-id, payload. Each line is auto-searchable/alertable. Design: never log secrets; log 'what happened around the request', add context; balance verbosity vs cost with sampling.

Introduction

What is this?

Libraries: pino, logrus, winston with pino, serilog; OpenTelemetry + logger bridge. Fields: ts, level, msg, service, trace_id, span_id, duration_ms, status, error/warn payload. Levels: debug/info/warn/error/fatal with env-variable gating. Multi-line export, log shaping middleware (request-id propagation). Pipelines: app → stdout → collector (filebeat/fluent-bit/OTel) → store (CloudWatch, Loki, Azure Log Analytics, GCP Logging) → queries/alerts/SIEM. Retention/tiering: hot (7-30d) / cold (30-90d) / archival (S3). Sampling: head (record % across the fleet) + tail (by result/error) to keep bill sane. Correlate with traces via trace-id.

Motivation

Why do you need it?

  • Wrecked debugging is 90% solved by 'which trace-id and what did the log say'.
  • Structured logs turn 'grep the string' into SQL-like queries and alerts.
  • Logs bridge the gap: metrics say latency; logs say which code path + why.
  • Good log hygiene is the cheapest forensic evidence you have.
Think about it

Mental model

Imagine

A pilot's flight log: every entry time-stamped, machine-readable, tied to the flight number

Think of

Structured logs

Pen-and-paper journals are searchable only by eye; structured flight logs link directly to lineage (trace-id) and can be queried on 'any hard landing after 3pm' — instantly.

Mechanics

How it works

  • Middleware injects a trace-id + request context into each log call.
  • Logger writes a single JSON line to stdout with that context.
  • Agent/collector ships to the store; queries and alert rules index it.
  • Sampling keeps volume bounded while preserving error/incident lines.
  • Retention per tier; SIEM cross-references adjacent sources.

Log pipeline.

App
Collector
Store
Query + alert
SIEM/archive
Hands-on

Examples

Structured log shape (pino-ish)

Structured log shape (pino-ish)
{"level":30,"time":1730000000000,"msg":"order failed",
 "service":"checkout","trace_id":"4f3a11...","duration_ms":84,
 "order_id":"o-123","error":{"code":"INV","detail":"out of stock"}}

Every field is queryable; the trace-id lets you pull the full request journey from the trace pillar.

Real world

In production

  • Request-scoped logging middleware everywhere (one standard context).
  • Alerting on structured log criteria (e.g., specific error code with rate).
  • Sampling: keep 100% of 5xx/errors; head-sample debug back to ~10%.
Guidelines

Best practices

  • Log facts, not feelings; include operation parameters where useful (never secrets/raw tokens).
  • Level-gate by environment; prod defaults to info-with-error-fill.
  • One JSON per line; avoid heredoc/multiline strings breaking parsers.
  • Set retention + tier by data class (audit/prod-events longest).
  • Correlate with traces: same trace-id in both pillars always.
Watch out

Common mistakes

  • Logging secrets/whole payloads (the incident-record you'd rather forget).
  • Textual ratt-tattle: too many log lines with zero structure or use.
  • Log aggregation with no indexing/retention tuning → big bill for noise.
  • Sampling 5xx related info out (you just thinned the forensic blood).
When things break

Debugging tips

  • Missing logs for a request → collector buffer or sampling; check trace-id-to-log join.
  • JSON parse errors in the sender → the store's parser saw multi-line/passthrough; fix emitter.
  • Too slow to query → add partition by service/time; avoid scans over the whole store.
  • Gaps in timeline correlate with collector downtime vs real silence — check the collector's state.
Interviews

Interview checkpoint

Practice

Hands-on lab

Emit, Collect, and Query Structured Logs
30 min · intermediate
In progress
Objectives
  • Emit JSON logs from a tiny service
  • Ship to a local store (Loki/Docker) or cloud
  • Run a correlation query with trace-id

Emit, Collect, and Query Structured Logs

30 min · intermediate

0/3 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

The killer feature of structured logs is:

2

When logging a failing request you should always include:

Go deeper

Official documentation

OpenTelemetry logs
docs
Loki docs
docs
CloudWatch Logs concepts
docs