Lesson 2 of 8 in Observability and incident response
Metrics with Prometheus: scraping and PromQL
Prometheus is a server that fetches numeric measurements from your services over HTTP at a fixed interval, called scraping, and stores them as time series. You ask questions about the stored data with its query language, PromQL.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Monitoring vs observability: metrics, logs, traces, Containers with Docker
The problem this solves
A metric is only useful if something collects it over time. One reading of a number tells you little, while the same number sampled every few seconds shows trends, spikes and the moment something changed.
Prometheus solves the collecting and storing part. Each service only has to publish its current numbers on a web page. Prometheus visits that page on a schedule, so a broken service cannot stop Prometheus from working, and a missing page is itself a sign of trouble.
The data model is built around a few metric types. A counter is a total that only goes up, such as requests served. A gauge is a value that moves both ways, such as queue depth. Choosing the right type decides which questions you can answer.
How it works
- Runs code
- Control
- Data
- People and clients
Prometheus requests the app's metrics page on a schedule, stores the samples, and answers PromQL queries from you.
- Prometheus to Your app: 1 GET /metrics
- Prometheus to Time series: 2 store samples
- You to Prometheus: 3 query
- Data
- Control
- People and clients
A counter is a growing total; the rate function turns two or more samples inside a time window into a per-second speed.
- Counter to rate(...[1m]): 1 samples
- rate(...[1m]) to sum by (status): 2 per-second
- sum by (status) to Graph: 3 draw
Prometheus starts every exchange. Your app never pushes anything. Every scrape interval, Prometheus sends a plain HTTP GET to the target's /metrics path, reads the text, and stores each number with the time it was read. In Kubernetes you rarely list targets by hand, because Prometheus can discover pods through the Kubernetes API.
Every stored number belongs to a series, identified by a metric name plus labels, for example app_requests_total{status="error"}. A label splits one metric into several series you can filter or add together.
A raw counter is rarely interesting, because its value depends on how long the process has run. rate(app_requests_total[1m]) instead gives requests per second averaged over the last minute, and it copes with a counter resetting to zero when the app restarts. A gauge is read as it is.
A histogram records how many observations fell under each upper bound, so you can estimate percentiles later. That is why latency is stored as buckets rather than a single average, which would hide slow outliers.
Try it yourself
Continue in ~/obs-lab. You will replace the single container with a Docker Compose file that runs your app, Prometheus and Grafana together. Grafana is used in a later concept. The lab stays up until the end of the postmortem concept.
- 1Stop the single container
docker stop obs-app
- 2Write the Prometheus config and the Compose file
cd ~/obs-lab mkdir -p prom cat > prom/prometheus.yml <<'EOF' global: scrape_interval: 5s scrape_configs: - job_name: demo-app static_configs: - targets: ["app:8000"] EOF cat > docker-compose.yml <<'EOF' services: app: image: python:3.12-slim command: sh -c "pip install -q prometheus-client && python -u /app/app.py" environment: ERROR_RATE: ${ERROR_RATE:-0.02} volumes: - ./app.py:/app/app.py:ro ports: - "8000:8000" prometheus: image: prom/prometheus volumes: - ./prom:/etc/prometheus:ro ports: - "9090:9090" grafana: image: grafana/grafana ports: - "3000:3000" EOF docker compose up -dThe target is
app:8000, notlocalhost:8000. Inside the Prometheus container,appis the name Compose gives the other service. Images are not pinned to versions because this is a throwaway lab. - 3Check that Prometheus can see the app
# open this address in your browser http://localhost:9090/targets
The
demo-apptarget should show state UP after the app finishes its pip install. A red DOWN with an error message tells you what Prometheus could not reach. - 4Run your first queries
# paste each into the Graph tab at http://localhost:9090/graph app_requests_total rate(app_requests_total[1m]) sum by (status) (rate(app_requests_total[1m])) app_queue_depth histogram_quantile(0.95, sum by (le) (rate(app_request_seconds_bucket[1m])))
Read them in order. The first is a raw total per status. The second is a per-second rate for each series. The third adds series together by status. The fourth is a gauge. The last estimates the 95th percentile latency from histogram buckets.
- 5Watch a counter reset
docker compose restart app
Graph
app_requests_totalagain. The totals fall back to zero when the process restarts, whilerate()over the same period stays sensible. This is why you query counters withrate()rather than reading them directly.
What breaks, and where to look
The target is DOWN with
Get "http://localhost:8000/metrics": dial tcp 127.0.0.1:8000: connect: connection refused.- WHY IT HAPPENS
- In
prometheus.ymlyou usedlocalhost. Inside the Prometheus container that means the Prometheus container itself, not your app. - CHECK FIRST
- Open
http://localhost:9090/targetsand read the exact URL it tried. Use the Compose service name (app:8000) as the target.
A query such as
rate(app_requests_total[5s])showsEmpty query result.- WHY IT HAPPENS
rate()needs at least two samples inside the window. With a 5 second scrape interval, a 5 second window often holds only one.- CHECK FIRST
- Make the range several times longer than the scrape interval, for example
[1m], and confirm the interval in the config.
You edited
prometheus.ymlbut Prometheus behaves as before, or the container keeps exiting.- WHY IT HAPPENS
- Prometheus reads its config at start. Either it was never reloaded, or the YAML has an error and Prometheus refused to start.
- CHECK FIRST
- Run
docker compose restart prometheus, thendocker compose logs prometheusand read the last lines for a config error.
When not to use this
- Do not put labels with unbounded values, such as user ids or request ids, on a metric. Every distinct value creates a new series and can overwhelm Prometheus. Put those details in logs.
- Prometheus keeps data on local disk for a limited time by default, and it is not an event store. Use it for numbers over time, and use logs for individual events.
Check yourself
Who starts the connection between Prometheus and your app, and why does that matter?
Prometheus does, by scraping. The app only publishes a page, so a failing app cannot break collection, and a target that stops answering shows up as DOWN.
Why query a counter with rate() instead of reading its value?
The raw total depends on process uptime and resets on restart. rate() gives a per-second speed and handles those resets.
When would you choose a gauge over a counter?
When the value moves both ways, such as queue depth or memory in use. Counters are for totals that only grow.
Keep going
Lesson 79 of 92 in the DevOps path