Lesson 2 of 8 in Observability and incident response

Metrics with Prometheus: scraping and PromQL

Core14 min

Prometheus is a server that fetches numeric measurements from your services over HTTP at a fixed interval, called scraping, and stores them as time series. You ask questions about the stored data with its query language, PromQL.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Monitoring vs observability: metrics, logs, traces, Containers with Docker

The problem this solves

A metric is only useful if something collects it over time. One reading of a number tells you little, while the same number sampled every few seconds shows trends, spikes and the moment something changed.

Prometheus solves the collecting and storing part. Each service only has to publish its current numbers on a web page. Prometheus visits that page on a schedule, so a broken service cannot stop Prometheus from working, and a missing page is itself a sign of trouble.

The data model is built around a few metric types. A counter is a total that only goes up, such as requests served. A gauge is a value that moves both ways, such as queue depth. Choosing the right type decides which questions you can answer.

How it works

Prometheus pulls from your app and you query it
Prometheus pulls from your app and you query itPrometheus requests the app's metrics page on a schedule, stores the samples, and answers PromQL queries from you.1 GET/metrics2 storesamples3 queryYour appserves /metricsPrometheusscrapes on a scheduleTime seriessamples on local diskYouask in PromQL
  • Runs code
  • Control
  • Data
  • People and clients

Prometheus requests the app's metrics page on a schedule, stores the samples, and answers PromQL queries from you.

  1. Prometheus to Your app: 1 GET /metrics
  2. Prometheus to Time series: 2 store samples
  3. You to Prometheus: 3 query
How a counter becomes a rate
How a counter becomes a rateA counter is a growing total; the rate function turns two or more samples inside a time window into a per-second speed.1 samples2 per-second3 drawCountertotal only goes uprate(...[1m])change per secondsum by (status)combine seriesGraphrequests per second
  • Data
  • Control
  • People and clients

A counter is a growing total; the rate function turns two or more samples inside a time window into a per-second speed.

  1. Counter to rate(...[1m]): 1 samples
  2. rate(...[1m]) to sum by (status): 2 per-second
  3. sum by (status) to Graph: 3 draw

Prometheus starts every exchange. Your app never pushes anything. Every scrape interval, Prometheus sends a plain HTTP GET to the target's /metrics path, reads the text, and stores each number with the time it was read. In Kubernetes you rarely list targets by hand, because Prometheus can discover pods through the Kubernetes API.

Every stored number belongs to a series, identified by a metric name plus labels, for example app_requests_total{status="error"}. A label splits one metric into several series you can filter or add together.

A raw counter is rarely interesting, because its value depends on how long the process has run. rate(app_requests_total[1m]) instead gives requests per second averaged over the last minute, and it copes with a counter resetting to zero when the app restarts. A gauge is read as it is.

A histogram records how many observations fell under each upper bound, so you can estimate percentiles later. That is why latency is stored as buckets rather than a single average, which would hide slow outliers.

Try it yourself

Continue in ~/obs-lab. You will replace the single container with a Docker Compose file that runs your app, Prometheus and Grafana together. Grafana is used in a later concept. The lab stays up until the end of the postmortem concept.

  1. 1
    Stop the single container
    docker stop obs-app
  2. 2
    Write the Prometheus config and the Compose file
    cd ~/obs-lab
    mkdir -p prom
    cat > prom/prometheus.yml <<'EOF'
    global:
      scrape_interval: 5s
    scrape_configs:
      - job_name: demo-app
        static_configs:
          - targets: ["app:8000"]
    EOF
    cat > docker-compose.yml <<'EOF'
    services:
      app:
        image: python:3.12-slim
        command: sh -c "pip install -q prometheus-client && python -u /app/app.py"
        environment:
          ERROR_RATE: ${ERROR_RATE:-0.02}
        volumes:
          - ./app.py:/app/app.py:ro
        ports:
          - "8000:8000"
      prometheus:
        image: prom/prometheus
        volumes:
          - ./prom:/etc/prometheus:ro
        ports:
          - "9090:9090"
      grafana:
        image: grafana/grafana
        ports:
          - "3000:3000"
    EOF
    docker compose up -d

    The target is app:8000, not localhost:8000. Inside the Prometheus container, app is the name Compose gives the other service. Images are not pinned to versions because this is a throwaway lab.

  3. 3
    Check that Prometheus can see the app
    # open this address in your browser
    http://localhost:9090/targets

    The demo-app target should show state UP after the app finishes its pip install. A red DOWN with an error message tells you what Prometheus could not reach.

  4. 4
    Run your first queries
    # paste each into the Graph tab at http://localhost:9090/graph
    app_requests_total
    rate(app_requests_total[1m])
    sum by (status) (rate(app_requests_total[1m]))
    app_queue_depth
    histogram_quantile(0.95, sum by (le) (rate(app_request_seconds_bucket[1m])))

    Read them in order. The first is a raw total per status. The second is a per-second rate for each series. The third adds series together by status. The fourth is a gauge. The last estimates the 95th percentile latency from histogram buckets.

  5. 5
    Watch a counter reset
    docker compose restart app

    Graph app_requests_total again. The totals fall back to zero when the process restarts, while rate() over the same period stays sensible. This is why you query counters with rate() rather than reading them directly.

What breaks, and where to look

  • The target is DOWN with Get "http://localhost:8000/metrics": dial tcp 127.0.0.1:8000: connect: connection refused.

    WHY IT HAPPENS
    In prometheus.yml you used localhost. Inside the Prometheus container that means the Prometheus container itself, not your app.
    CHECK FIRST
    Open http://localhost:9090/targets and read the exact URL it tried. Use the Compose service name (app:8000) as the target.
  • A query such as rate(app_requests_total[5s]) shows Empty query result.

    WHY IT HAPPENS
    rate() needs at least two samples inside the window. With a 5 second scrape interval, a 5 second window often holds only one.
    CHECK FIRST
    Make the range several times longer than the scrape interval, for example [1m], and confirm the interval in the config.
  • You edited prometheus.yml but Prometheus behaves as before, or the container keeps exiting.

    WHY IT HAPPENS
    Prometheus reads its config at start. Either it was never reloaded, or the YAML has an error and Prometheus refused to start.
    CHECK FIRST
    Run docker compose restart prometheus, then docker compose logs prometheus and read the last lines for a config error.

When not to use this

  • Do not put labels with unbounded values, such as user ids or request ids, on a metric. Every distinct value creates a new series and can overwhelm Prometheus. Put those details in logs.
  • Prometheus keeps data on local disk for a limited time by default, and it is not an event store. Use it for numbers over time, and use logs for individual events.

Check yourself

Who starts the connection between Prometheus and your app, and why does that matter?

Prometheus does, by scraping. The app only publishes a page, so a failing app cannot break collection, and a target that stops answering shows up as DOWN.

Why query a counter with rate() instead of reading its value?

The raw total depends on process uptime and resets on restart. rate() gives a per-second speed and handles those resets.

When would you choose a gauge over a counter?

When the value moves both ways, such as queue depth or memory in use. Counters are for totals that only grow.

Lesson 79 of 92 in the DevOps path

Search

Search cheat sheets and projects