Lesson 1 of 8 in Observability and incident response

Monitoring vs observability: metrics, logs, traces

Core10 min

Monitoring is watching a set of measurements you chose in advance to tell whether a system is healthy. Observability is how well you can work out what is happening inside a system from the data it emits, including for problems nobody predicted. The three common kinds of data, called signals, are metrics, logs and traces.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Containers with Docker

The problem this solves

A service running on a server you cannot see into is a black box. When users report errors, you have only guesses unless the service has been recording what it does.

Some failures are ones you saw coming, such as a full disk, and a check prepared in advance catches them. Many are new. Observability is the habit of emitting varied, detailed data so that during a failure you can ask a question you did not plan for without deploying new code first.

Each signal answers a different question. Metrics say how much and how often. Logs say what happened, in detail. Traces say where one request spent its time as it crossed several services.

How it works

One service emits three kinds of data
One service emits three kinds of dataA running service produces metrics, logs and traces, and each one answers a different question about what the service is doing.how much, how oftenwhat happenedwhere time wentYour servicerunning in a containerMetricsnumbers over timeLogsevents with detailTracesone request's path
  • Runs code
  • Data

A running service produces metrics, logs and traces, and each one answers a different question about what the service is doing.

  1. Your service to Metrics: how much, how often
  2. Your service to Logs: what happened
  3. Your service to Traces: where time went
A common order for using the signals
A common order for using the signalsA metric alert tells you something is wrong, a trace shows which step of a request is the problem, and the logs of that step explain why.1 find thestep2 read thedetail3 actMetric alerterror rate is upTracewhich step is slowLogswhy that step failedYou decidemitigate or fix
  • Control
  • Other
  • People and clients

A metric alert tells you something is wrong, a trace shows which step of a request is the problem, and the logs of that step explain why.

  1. Metric alert to Trace: 1 find the step
  2. Trace to Logs: 2 read the detail
  3. Logs to You decide: 3 act

Your service is the source. It does its normal work and, as a side effect, it counts things (metrics), writes down events (logs) and marks the start and end of operations (traces).

Metrics are cheap to store and good for spotting that something changed. They cannot tell you which single request failed. Logs can, but they are bulky and hard to search without structure. Traces connect the dots across services but only exist if you instrument for them.

In the second picture, the order is a habit rather than a rule. Start wide with a metric that says users are affected, narrow to the step with a trace, then read the logs of that step. Many teams skip traces and go straight from metrics to logs, which works fine for a single service.

Try it yourself

This lab needs Docker (on Windows, use WSL). It starts a tiny Python app that pretends to handle requests, exposes metrics on port 8000 and writes JSON logs. Everything lives in one throwaway folder, ~/obs-lab.

  1. 1
    Create the lab folder and the app
    mkdir -p ~/obs-lab && cd ~/obs-lab
    cat > app.py <<'EOF'
    import json, os, random, time, uuid
    from prometheus_client import Counter, Gauge, Histogram, start_http_server
    
    REQUESTS = Counter("app_requests", "Requests handled", ["status"])
    LATENCY = Histogram("app_request_seconds", "Request duration in seconds")
    QUEUE = Gauge("app_queue_depth", "Jobs waiting")
    ERROR_RATE = float(os.environ.get("ERROR_RATE", "0.02"))
    
    # create both series at zero so queries work before the first error
    REQUESTS.labels(status="ok")
    REQUESTS.labels(status="error")
    
    start_http_server(8000)
    while True:
        request_id = uuid.uuid4().hex[:8]
        start = time.time()
        time.sleep(random.uniform(0.01, 0.2))
        failed = random.random() < ERROR_RATE
        elapsed = time.time() - start
        LATENCY.observe(elapsed)
        REQUESTS.labels(status="error" if failed else "ok").inc()
        QUEUE.set(random.randint(0, 20))
        if failed:
            print(json.dumps({"level": "error", "msg": "payment failed", "request_id": request_id, "duration_ms": round(elapsed * 1000)}), flush=True)
        elif random.random() < 0.05:
            print(json.dumps({"level": "info", "msg": "request ok", "request_id": request_id, "duration_ms": round(elapsed * 1000)}), flush=True)
    EOF

    The app handles a few requests per second. About 2 percent fail, so there is always something to look at. The file is written by the heredoc; nothing outside ~/obs-lab is touched.

  2. 2
    Run it in a container
    docker run -d --rm --name obs-app -p 8000:8000 \
      -v "$PWD/app.py:/app/app.py:ro" \
      python:3.12-slim sh -c "pip install -q prometheus-client && python -u /app/app.py"

    The container installs one small library first, so wait about 15 seconds before the next step.

  3. 3
    Look at the logs signal
    docker logs --tail 5 obs-app

    Each line is one JSON object with a level, a message and a request_id. Only errors and a sample of successes are logged, so a quiet moment can show fewer than five lines.

  4. 4
    Look at the metrics signal
    curl -s localhost:8000/metrics | grep "^app_"

    This is the raw text a metrics server reads. You see a counter app_requests_total per status, a gauge app_queue_depth and histogram buckets named app_request_seconds_bucket. Run it twice a few seconds apart and compare the counters.

What breaks, and where to look

  • curl: (7) Failed to connect to localhost port 8000 right after starting the container.

    WHY IT HAPPENS
    The container is still running pip install and has not started the metrics server yet.
    CHECK FIRST
    Run docker logs obs-app. Wait until the pip output stops, then retry the curl.
  • Bind for 0.0.0.0:8000 failed: port is already allocated

    WHY IT HAPPENS
    Another container or program already uses host port 8000, often an earlier run of this lab.
    CHECK FIRST
    Run docker ps to find the old container and docker stop obs-app to remove it.
  • Every dashboard is green, yet users say the site is broken.

    WHY IT HAPPENS
    Monitoring only watches what you chose to measure. The failing behaviour may not be measured at all, or may be measured from inside the service rather than from where users stand.
    CHECK FIRST
    Use the product the way a user does. Whatever you notice that no panel shows is the missing signal to add.

When not to use this

  • For one small app, structured logs plus a simple uptime check may be all you need. A full metrics, logs and traces stack costs time to run and understand.
  • Observability data shortens the time to understand a problem. It does not make a system reliable by itself.

Check yourself

Which signal would you use to see that the error rate rose at 14:05, and which to see the exact request that failed?

Metrics show the rate change over time. Logs, or a trace, show the specific failed request and its detail.

What is the difference between monitoring and observability?

Monitoring checks measurements chosen in advance. Observability is having enough varied data to explain behaviour you did not predict.

Why does the app log only errors and a sample of successes?

Logging every success is bulky and costly to store and search. Errors matter most, and a sample is enough to see normal behaviour.

Lesson 78 of 92 in the DevOps path

Search

Search cheat sheets and projects