Lesson 3 of 8 in Observability and incident response

Logs that help: structure, levels, correlation IDs

Core9 min

A log is a time-ordered record of events a program chooses to write down. Structured logging means each line has named fields, usually JSON, so tools can filter on them. A correlation ID is a value, such as a request ID, written on every line about the same request so you can follow it.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Monitoring vs observability: metrics, logs, traces

The problem this solves

Free-form log lines like payment failed for order are easy to write and hard to use. To find all failures for one customer you end up writing fragile text patterns, and any change in wording breaks them.

Two more problems appear with scale. Without levels, the one line that matters is buried among thousands of routine ones. Without a shared identifier, a request that touched three services leaves three unconnected trails.

Structure, levels and a correlation ID are cheap habits that fix all three. They cost a few extra fields per line and save the time you would spend guessing during an outage.

How it works

From a log line to a search
From a log line to a searchThe app writes structured lines to stdout, an agent collects them, a log store such as Loki keeps them, and you search by field.1 emit lines2 ship withlabels3 queryYour appwrites JSON to stdoutLog agentcollects and labelsLog storefor example LokiYousearch by field
  • Runs code
  • Control
  • Data
  • People and clients

The app writes structured lines to stdout, an agent collects them, a log store such as Loki keeps them, and you search by field.

  1. Your app to Log agent: 1 emit lines
  2. Log agent to Log store: 2 ship with labels
  3. You to Log store: 3 query
One request ID across services
One request ID across servicesThe first service creates a request ID and passes it on, so both services write it on their lines and one search finds the whole story.1 request2 pass the idon3 lines share the idClientsends a requestService Acreates request_idService Blogs the same idSearchable logsfilter on request_id
  • People and clients
  • Runs code
  • Data

The first service creates a request ID and passes it on, so both services write it on their lines and one search finds the whole story.

  1. Client to Service A: 1 request
  2. Service A to Service B: 2 pass the id on
  3. Service B to Searchable logs: 3 lines share the id

Write one event per line, as an object with a few stable fields: a level, a short message, and the identifiers that matter such as request_id. Keep the message text constant and put the changing parts in fields, so the same event always looks the same.

Levels rank importance. error means something failed that someone may need to act on, warn means odd but handled, info records normal milestones, and debug is detail you normally leave off. Setting the minimum level controls the volume.

A correlation ID ties lines together. The first service to see a request creates the ID, and every call it makes passes the ID along, usually in an HTTP header. Each service then adds the ID to every line it writes for that request.

Loki is a log store from Grafana. It does not index the full text of lines. It indexes labels, such as the app name, and stores compressed log data, which keeps it cheap. You find a stream with labels first, then filter inside it with LogQL, a query language modelled on PromQL.

Try it yourself

Use the lab from the Prometheus concept, still running in ~/obs-lab. These steps only read logs; they change nothing.

  1. 1
    Read recent log lines
    cd ~/obs-lab
    docker compose logs --no-log-prefix --tail 10 app

    Each line is JSON with level, msg, request_id and duration_ms.

  2. 2
    Filter by level
    docker compose logs --no-log-prefix app | grep '"level": "error"' | tail -3

    Because the level is a named field in a fixed format, a simple filter finds exactly the errors.

  3. 3
    Follow one request by its ID
    # copy a request_id from an error line above, then:
    docker compose logs --no-log-prefix app | grep '<request-id>'

    Replace the placeholder with a real ID. In this one-service lab the search returns a single line. With several services logging the same ID, it would return the whole path of the request.

  4. 4
    Count errors
    docker compose logs --no-log-prefix app | grep -c '"level": "error"'

    Counting log lines works for a quick check. For ongoing numbers, a metric is cheaper, which is why the app also keeps a counter.

What breaks, and where to look

  • The container is running and the app is busy, but docker compose logs app shows nothing, or lines arrive minutes late.

    WHY IT HAPPENS
    Python buffers its output when stdout is not a terminal. The lines sit in a buffer until it fills.
    CHECK FIRST
    Run Python with -u, set PYTHONUNBUFFERED=1, or pass flush=True to print, as the lab app does.
  • You search for a request ID and only one service has lines for it.

    WHY IT HAPPENS
    The ID was not passed on to the next service, or the services name the field differently, such as request_id in one and reqId in another.
    CHECK FIRST
    Search each service's logs for the ID separately, and look at the outgoing request headers to see whether the ID is forwarded.
  • The disk on the host fills up and the cause is container logs.

    WHY IT HAPPENS
    Docker's default json-file log driver does not rotate logs unless you set limits, so a chatty service grows its log file without bound.
    CHECK FIRST
    Run docker inspect --format '{{.HostConfig.LogConfig}}' <container-name> to see the driver and options, and set max-size and max-file if they are missing.

When not to use this

  • Never write secrets, passwords, tokens or full card numbers into logs. Logs are copied to many places and kept for a long time.
  • Do not count things by grepping logs in production. A counter metric answers 'how many' cheaper and faster, and logs are better for 'which one and why'.

Check yourself

Why keep the message text constant and put changing values in fields?

The same event then always looks identical, so one filter finds every occurrence, and the varying parts can be searched as fields.

What does a correlation ID let you do?

Find every log line, across all services, that belongs to one request by searching for a single value.

How does Loki keep storage cheap compared with indexing full text?

It indexes only labels and stores the log content compressed, then filters the content at query time inside the streams you selected.

Lesson 80 of 92 in the DevOps path

Search

Search cheat sheets and projects