Monitoring
Intermediate 16 min read Monitoring

Logging at Scale

Logs are the record of what happened. Learn structured logging, log shipping with Filebeat, the ELK/EFK pipeline, correlation IDs that tie a request together, and index lifecycle policies that keep storage sane.

structured loggingELKEFKFilebeatLogstashElasticsearchKibanacorrelation IDILMlog retention
Prerequisites: Observability Fundamentals Metrics & Prometheus
Introduction

What is this?

Logging at scale means turning a per-host grep into a searchable, centralized record. Structured logging writes every event as machine-parseable key-value pairs, typically JSON. A shipper like Filebeat tails logs and forwards them to a pipeline, Logstash parses and enriches, Elasticsearch indexes the documents, and Kibana provides search and visualization — the ELK stack, or EFK when Fluentd or Fluent Bit replaces Logstash. Correlation IDs and index lifecycle management (ILM) complete the picture.

Motivation

Why do you need it?

  • Grep across files breaks beyond a handful of hosts; a central store gives one searchable pane of glass.
  • Structured logs are queryable — filter by level, service and trace_id in milliseconds instead of scanning text.
  • A correlation ID turns thousands of interleaved log lines into the story of one request.
  • Centralized logs provide the audit trail and historical record that dashboards and traces do not keep.
  • Index lifecycle management keeps cost and search performance under control as volume grows.
Think about it

Mental model

Imagine

A public library

Think of

A centralized log store

A shoebox of receipts is a single service's raw log file — you can find something if you remember when it happened. A library is Elasticsearch: every receipt is a record with fields, indexed by author, date and subject, so you can search across all of them instantly. Logstash is the librarian who stamps and categorizes each new record as it arrives.

Mechanics

How it works

  • Applications emit one JSON object per event to stdout or a log file, with a fixed schema: timestamp, level, message, service and trace_id.
  • A shipper agent (Filebeat) tails the files, tracks offsets in a registry, and forwards events to the pipeline over a reliable transport.
  • Logstash (or a lightweight pipeline like Fluent Bit) parses, normalizes fields, enriches with metadata and redacts sensitive values.
  • Elasticsearch indexes each document by field, mapping strings as keyword or text, and creates one index per time period.
  • Kibana queries the index using the Query DSL, and ILM rolls, shrinks, deletes or moves old indices to warm or cold tiers.

The ELK/EFK pipeline from application to dashboard.

Application
Filebeat
Logstash
Elasticsearch
Kibana
Engineer
Reference

Key commands

docker logs --tail 50 --since 5m myapp

Read the last 50 lines from a container's logs emitted in the last 5 minutes.

curl -s 'http://localhost:9200/_cat/indices?v'

List all indices with their health, size and document counts.

curl -s 'http://localhost:9200/logs-2026.08.08/_search?q=level:error'

Search a specific day's index for error-level events.

curl -s 'http://localhost:9200/logs-*/_search?q=msg:"payment failed"'

Search across all daily indices for an exact message phrase.

filebeat modules list

Show the built-in Filebeat modules available for common systems.

Hands-on

Examples

A structured log event and how you query it

A structured log event and how you query it
{"timestamp":"2026-08-08T10:11:12.345Z","level":"error","service":"checkout","trace_id":"a1b2c3d4","user_id":"u-991","msg":"payment failed","http.status":503,"duration_ms":1430}
 
# Search every service for that one request's error
curl -s 'http://localhost:9200/logs-*/_search?q=service:checkout%20AND%20level:error%20AND%20trace_id:a1b2c3d4'
 
# List the daily indices being rolled
curl -s 'http://localhost:9200/_cat/indices?v' | head -n 8

Each log event is one JSON line, so parsers, exporters and query tools can treat fields as first-class values. The query targets a date-based index pattern, and _cat/indices shows the lifecycle at work: one index per day is rolled, then moved or deleted by the ILM policy.

Real world

In production

  • The checkout service logs trace_id on every event, so a support ticket number maps straight to a searchable request story.
  • A Filebeat-to-Logstash-to-Elasticsearch pipeline normalizes fields across 40 services with differing log conventions.
  • An ILM policy keeps hot indices for 7 days, warm for 30, and deletes after 90, so costs stay predictable.
  • Audit logs are routed to a separate index with longer retention and restricted access for compliance.
  • PII and token redaction runs in the pipeline so raw payloads never reach Elasticsearch or support dashboards.
Guidelines

Best practices

  • Log in JSON with a fixed schema: timestamp, level, message, service and trace_id on every line.
  • Never log secrets, tokens, or full request payloads; redact at the source and again in the pipeline.
  • Propagate a correlation ID from the edge of the request so all services log the same value.
  • Log at INFO by default and gate DEBUG behind a configuration flag to control volume.
  • Match retention to value: keep error and audit logs long, and debug logs short.
Watch out

Common mistakes

  • Logging unstructured free text, which forces grep-and-hope debugging and makes queries useless.
  • Logging secrets or request bodies that end up in the support queue or a security review.
  • Spreading one logical event across multiple lines, breaking line-oriented parsers.
  • Using inconsistent timestamp formats and timezones, breaking time-range searches.
  • Shipping logs with no retention plan until disk fills and indexing stops.
Root cause analysis

Common issues

Error
Logs stop appearing in Kibana
Root cause

The shipper crashed, the pipeline is down, or the daily index failed to roll.

Fix

Check Filebeat logs and the registry state, confirm Elasticsearch is healthy, and restart the shipper.

Error
Events are indexed but searches return nothing
Root cause

A field is mapped as text while the query uses keyword syntax, or the query targets the wrong index.

Fix

Inspect the mapping with /_mapping, use the .keyword variant for exact matches, and verify the index pattern.

Error
Disk fills up on the log hosts
Root cause

No rotation on the app side and no ILM policy on the cluster.

Fix

Enable file rotation, apply an ILM policy with rollover and deletion, and cap shipper file descriptors.

When things break

Debugging tips

  • Filter by trace_id first — it reconstructs the whole request path across every service.
  • Compare error counts before and after a deploy by grouping on service and level.
  • When logs stop appearing, check the shipper's registry and status before suspecting the app.
  • Use _cat/indices to confirm daily indices are being created and rolled by ILM.
  • Verify timestamps are ISO-8601 UTC so Kibana time filters behave consistently.
Interviews

Interview checkpoint

Practice

Hands-on lab

Index a Structured Log Event in Elasticsearch
25 min · intermediate
In progress
Objectives
  • Generate a structured JSON log event
  • Run a single-node Elasticsearch cluster
  • Index a log document with curl
  • Search the indexed document by field

Index a Structured Log Event in Elasticsearch

25 min · intermediate

0/7 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which component ingests, parses and normalizes logs in the ELK stack?

2

What is the primary benefit of a correlation ID?

3

Filebeat ships logs, and Elasticsearch is the search and analytics engine.

Quick reference

Cheat sheet

Search Elasticsearch
GET /_cat/indices?vList indices and their status
_search?q=level:errorQuery string search for a field
_search?q=trace_id:abc123Correlate one request's logs
GET /<index>/_mappingInspect field types (keyword vs text)
GET /_cat/ilm/policyShow index lifecycle policies
Get logs locally
docker logs --tail 50 myappTail a container's logs
journalctl -u myapp -fFollow a systemd service's logs
tail -f /var/log/app.logWatch a log file grow
kubectl logs -f deploy/myappStream pod logs in Kubernetes
grep 'trace_id=abc' app.logFilter lines for one request
Go deeper

Official documentation

Elasticsearch Reference
docs
Filebeat Reference
docs
Logstash Reference
docs
Google — Structured Logging Best Practices
docs