Lesson 1 of 8 in Observability and incident response
Monitoring vs observability: metrics, logs, traces
Monitoring is watching a set of measurements you chose in advance to tell whether a system is healthy. Observability is how well you can work out what is happening inside a system from the data it emits, including for problems nobody predicted. The three common kinds of data, called signals, are metrics, logs and traces.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Containers with Docker
The problem this solves
A service running on a server you cannot see into is a black box. When users report errors, you have only guesses unless the service has been recording what it does.
Some failures are ones you saw coming, such as a full disk, and a check prepared in advance catches them. Many are new. Observability is the habit of emitting varied, detailed data so that during a failure you can ask a question you did not plan for without deploying new code first.
Each signal answers a different question. Metrics say how much and how often. Logs say what happened, in detail. Traces say where one request spent its time as it crossed several services.
How it works
- Runs code
- Data
A running service produces metrics, logs and traces, and each one answers a different question about what the service is doing.
- Your service to Metrics: how much, how often
- Your service to Logs: what happened
- Your service to Traces: where time went
- Control
- Other
- People and clients
A metric alert tells you something is wrong, a trace shows which step of a request is the problem, and the logs of that step explain why.
- Metric alert to Trace: 1 find the step
- Trace to Logs: 2 read the detail
- Logs to You decide: 3 act
Your service is the source. It does its normal work and, as a side effect, it counts things (metrics), writes down events (logs) and marks the start and end of operations (traces).
Metrics are cheap to store and good for spotting that something changed. They cannot tell you which single request failed. Logs can, but they are bulky and hard to search without structure. Traces connect the dots across services but only exist if you instrument for them.
In the second picture, the order is a habit rather than a rule. Start wide with a metric that says users are affected, narrow to the step with a trace, then read the logs of that step. Many teams skip traces and go straight from metrics to logs, which works fine for a single service.
Try it yourself
This lab needs Docker (on Windows, use WSL). It starts a tiny Python app that pretends to handle requests, exposes metrics on port 8000 and writes JSON logs. Everything lives in one throwaway folder, ~/obs-lab.
- 1Create the lab folder and the app
mkdir -p ~/obs-lab && cd ~/obs-lab cat > app.py <<'EOF' import json, os, random, time, uuid from prometheus_client import Counter, Gauge, Histogram, start_http_server REQUESTS = Counter("app_requests", "Requests handled", ["status"]) LATENCY = Histogram("app_request_seconds", "Request duration in seconds") QUEUE = Gauge("app_queue_depth", "Jobs waiting") ERROR_RATE = float(os.environ.get("ERROR_RATE", "0.02")) # create both series at zero so queries work before the first error REQUESTS.labels(status="ok") REQUESTS.labels(status="error") start_http_server(8000) while True: request_id = uuid.uuid4().hex[:8] start = time.time() time.sleep(random.uniform(0.01, 0.2)) failed = random.random() < ERROR_RATE elapsed = time.time() - start LATENCY.observe(elapsed) REQUESTS.labels(status="error" if failed else "ok").inc() QUEUE.set(random.randint(0, 20)) if failed: print(json.dumps({"level": "error", "msg": "payment failed", "request_id": request_id, "duration_ms": round(elapsed * 1000)}), flush=True) elif random.random() < 0.05: print(json.dumps({"level": "info", "msg": "request ok", "request_id": request_id, "duration_ms": round(elapsed * 1000)}), flush=True) EOFThe app handles a few requests per second. About 2 percent fail, so there is always something to look at. The file is written by the heredoc; nothing outside
~/obs-labis touched. - 2Run it in a container
docker run -d --rm --name obs-app -p 8000:8000 \ -v "$PWD/app.py:/app/app.py:ro" \ python:3.12-slim sh -c "pip install -q prometheus-client && python -u /app/app.py"
The container installs one small library first, so wait about 15 seconds before the next step.
- 3Look at the logs signal
docker logs --tail 5 obs-app
Each line is one JSON object with a level, a message and a request_id. Only errors and a sample of successes are logged, so a quiet moment can show fewer than five lines.
- 4Look at the metrics signal
curl -s localhost:8000/metrics | grep "^app_"
This is the raw text a metrics server reads. You see a counter
app_requests_totalper status, a gaugeapp_queue_depthand histogram buckets namedapp_request_seconds_bucket. Run it twice a few seconds apart and compare the counters.
What breaks, and where to look
curl: (7) Failed to connect to localhost port 8000right after starting the container.- WHY IT HAPPENS
- The container is still running
pip installand has not started the metrics server yet. - CHECK FIRST
- Run
docker logs obs-app. Wait until the pip output stops, then retry the curl.
Bind for 0.0.0.0:8000 failed: port is already allocated- WHY IT HAPPENS
- Another container or program already uses host port 8000, often an earlier run of this lab.
- CHECK FIRST
- Run
docker psto find the old container anddocker stop obs-appto remove it.
Every dashboard is green, yet users say the site is broken.
- WHY IT HAPPENS
- Monitoring only watches what you chose to measure. The failing behaviour may not be measured at all, or may be measured from inside the service rather than from where users stand.
- CHECK FIRST
- Use the product the way a user does. Whatever you notice that no panel shows is the missing signal to add.
When not to use this
- For one small app, structured logs plus a simple uptime check may be all you need. A full metrics, logs and traces stack costs time to run and understand.
- Observability data shortens the time to understand a problem. It does not make a system reliable by itself.
Check yourself
Which signal would you use to see that the error rate rose at 14:05, and which to see the exact request that failed?
Metrics show the rate change over time. Logs, or a trace, show the specific failed request and its detail.
What is the difference between monitoring and observability?
Monitoring checks measurements chosen in advance. Observability is having enough varied data to explain behaviour you did not predict.
Why does the app log only errors and a sample of successes?
Logging every success is bulky and costly to store and search. Errors matter most, and a sample is enough to see normal behaviour.
Lesson 78 of 92 in the DevOps path