Lesson 3 of 8 in Observability and incident response
Logs that help: structure, levels, correlation IDs
A log is a time-ordered record of events a program chooses to write down. Structured logging means each line has named fields, usually JSON, so tools can filter on them. A correlation ID is a value, such as a request ID, written on every line about the same request so you can follow it.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Monitoring vs observability: metrics, logs, traces
The problem this solves
Free-form log lines like payment failed for order are easy to write and hard to use. To find all failures for one customer you end up writing fragile text patterns, and any change in wording breaks them.
Two more problems appear with scale. Without levels, the one line that matters is buried among thousands of routine ones. Without a shared identifier, a request that touched three services leaves three unconnected trails.
Structure, levels and a correlation ID are cheap habits that fix all three. They cost a few extra fields per line and save the time you would spend guessing during an outage.
How it works
- Runs code
- Control
- Data
- People and clients
The app writes structured lines to stdout, an agent collects them, a log store such as Loki keeps them, and you search by field.
- Your app to Log agent: 1 emit lines
- Log agent to Log store: 2 ship with labels
- You to Log store: 3 query
- People and clients
- Runs code
- Data
The first service creates a request ID and passes it on, so both services write it on their lines and one search finds the whole story.
- Client to Service A: 1 request
- Service A to Service B: 2 pass the id on
- Service B to Searchable logs: 3 lines share the id
Write one event per line, as an object with a few stable fields: a level, a short message, and the identifiers that matter such as request_id. Keep the message text constant and put the changing parts in fields, so the same event always looks the same.
Levels rank importance. error means something failed that someone may need to act on, warn means odd but handled, info records normal milestones, and debug is detail you normally leave off. Setting the minimum level controls the volume.
A correlation ID ties lines together. The first service to see a request creates the ID, and every call it makes passes the ID along, usually in an HTTP header. Each service then adds the ID to every line it writes for that request.
Loki is a log store from Grafana. It does not index the full text of lines. It indexes labels, such as the app name, and stores compressed log data, which keeps it cheap. You find a stream with labels first, then filter inside it with LogQL, a query language modelled on PromQL.
Try it yourself
Use the lab from the Prometheus concept, still running in ~/obs-lab. These steps only read logs; they change nothing.
- 1Read recent log lines
cd ~/obs-lab docker compose logs --no-log-prefix --tail 10 app
Each line is JSON with
level,msg,request_idandduration_ms. - 2Filter by level
docker compose logs --no-log-prefix app | grep '"level": "error"' | tail -3
Because the level is a named field in a fixed format, a simple filter finds exactly the errors.
- 3Follow one request by its ID
# copy a request_id from an error line above, then: docker compose logs --no-log-prefix app | grep '<request-id>'
Replace the placeholder with a real ID. In this one-service lab the search returns a single line. With several services logging the same ID, it would return the whole path of the request.
- 4Count errors
docker compose logs --no-log-prefix app | grep -c '"level": "error"'
Counting log lines works for a quick check. For ongoing numbers, a metric is cheaper, which is why the app also keeps a counter.
What breaks, and where to look
The container is running and the app is busy, but
docker compose logs appshows nothing, or lines arrive minutes late.- WHY IT HAPPENS
- Python buffers its output when stdout is not a terminal. The lines sit in a buffer until it fills.
- CHECK FIRST
- Run Python with
-u, setPYTHONUNBUFFERED=1, or passflush=Truetoprint, as the lab app does.
You search for a request ID and only one service has lines for it.
- WHY IT HAPPENS
- The ID was not passed on to the next service, or the services name the field differently, such as
request_idin one andreqIdin another. - CHECK FIRST
- Search each service's logs for the ID separately, and look at the outgoing request headers to see whether the ID is forwarded.
The disk on the host fills up and the cause is container logs.
- WHY IT HAPPENS
- Docker's default
json-filelog driver does not rotate logs unless you set limits, so a chatty service grows its log file without bound. - CHECK FIRST
- Run
docker inspect --format '{{.HostConfig.LogConfig}}' <container-name>to see the driver and options, and setmax-sizeandmax-fileif they are missing.
When not to use this
- Never write secrets, passwords, tokens or full card numbers into logs. Logs are copied to many places and kept for a long time.
- Do not count things by grepping logs in production. A counter metric answers 'how many' cheaper and faster, and logs are better for 'which one and why'.
Check yourself
Why keep the message text constant and put changing values in fields?
The same event then always looks identical, so one filter finds every occurrence, and the varying parts can be searched as fields.
What does a correlation ID let you do?
Find every log line, across all services, that belongs to one request by searching for a single value.
How does Loki keep storage cheap compared with indexing full text?
It indexes only labels and stores the log content compressed, then filters the content at query time inside the streams you selected.
Keep going
Lesson 80 of 92 in the DevOps path