Monitoring
Intermediate 16 min read Monitoring

Metrics & Prometheus

Prometheus is the de facto standard for collecting and querying metrics. Master the pull model, the four metric types, labels, retention, and enough PromQL to answer real questions about your systems.

PrometheusPromQLcountergaugehistogramsummarylabelsscraperetentionrecording rules
Prerequisites: Observability Fundamentals
Introduction

What is this?

Prometheus is a pull-based time series database and monitoring toolkit. On a fixed interval it scrapes HTTP endpoints that expose metrics in a text format, stores each unique label set as its own series, and answers queries with PromQL. Four metric types encode meaning: counters only increase, gauges go up and down, histograms count observations in pre-defined buckets, and summaries expose client-computed quantiles.

Motivation

Why do you need it?

  • The pull model fits containers and Kubernetes naturally: Prometheus discovers targets and immediately knows when one disappears.
  • A label-based data model makes queries flexible — you can slice, aggregate and join on any dimension.
  • Choosing counter, gauge, histogram or summary forces you to think about what the metric actually means.
  • PromQL is the industry-standard metric language, also implemented by Thanos, VictoriaMetrics, Grafana Mimir and AWS Managed Prometheus.
  • Local storage, service discovery and config-file operation make it easy to self-host with predictable retention.
Think about it

Mental model

Imagine

A car's instruments

Think of

Prometheus metric types

The trip odometer is a counter — it only goes up and resets at zero. The speedometer is a gauge — it moves in both directions. A histogram is like tallying passengers into height buckets (under 1.5m, under 1.8m, under 2.1m); a summary is like someone pre-computing the median height for you.

Mechanics

How it works

  • Each target exposes a /metrics endpoint in the text exposition format: comment lines starting with # HELP and # TYPE, then one sample per line with optional label sets.
  • Prometheus pulls every target on its scrape_interval using HTTP; a failed scrape is recorded as the up metric being 0 for that target.
  • Every unique combination of metric name and label values becomes one time series; samples are appended to a block-structured storage engine.
  • PromQL resolves instant vectors (one value per series right now) and range vectors (values over a window) for functions like rate and histogram_quantile.
  • Recording rules precompute expensive or repeated expressions, and alerting rules evaluate threshold conditions on the same evaluation schedule.

How Prometheus scrapes, stores and serves metrics.

Targets
Scrape job
Store
PromQL
Consumers
Engineer
Reference

Key commands

curl -s http://localhost:9090/metrics | head

Read the exposition format directly from a target's /metrics endpoint.

curl -s http://localhost:9090/api/v1/targets

List configured scrape targets and their current health state.

curl -s 'http://localhost:9090/api/v1/query?query=up'

Run an instant PromQL query through the HTTP API.

promtool check config prometheus.yml

Validate the Prometheus configuration before reloading it.

promtool check rules rules.yml

Validate recording and alerting rule files for syntax errors.

Hands-on

Examples

Query a counter correctly with rate()

Query a counter correctly with rate()
# WRONG: a raw counter just grows since the last restart
http_requests_total{job="web"}   →  1284773
 
# RIGHT: rate() converts counter growth to requests per second
rate(http_requests_total{job="web"}[5m])          →  42.5
 
# Errors per second, sliced by instance
sum by (instance) (rate(http_requests_total{job="web", status=~"5.."}[5m]))
 
# p99 latency from a histogram across all instances
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

A counter is monotonically increasing, so a raw value is only meaningful combined with a window. rate() computes per-second growth over that window and properly handles counter resets. Histograms aggregate cleanly with sum by (le) before histogram_quantile picks the bucket boundary containing the 99th percentile.

Real world

In production

  • The kube-prometheus stack scrapes node metrics, kubelet metrics and application endpoints using relabeling to discover pods.
  • SLO burn-rate alerts run recording rules against high-resolution counters and evaluate them every 30 seconds.
  • Latency SLOs are computed from histograms because, unlike summaries, histograms aggregate across instances.
  • Remote write or federation ships series from edge clusters to a central long-term store while each cluster keeps local Prometheus.
  • Cardinality is audited with the TSDB stats tools so series growth stays inside the capacity budget.
Guidelines

Best practices

  • Use counters only for monotonically increasing totals and always pass them through rate() or increase().
  • Prefer histograms over summaries for latency: summaries cannot be aggregated across instances, histograms can.
  • Keep label cardinality bounded — never embed request IDs, user IDs or IP addresses as label values.
  • Follow metric naming conventions: namespace_name_unit_total, e.g. http_requests_total or node_cpu_seconds_total.
  • Put expensive or repeated queries in recording rules; keep alerting rules simple and readable.
Watch out

Common mistakes

  • Plotting a raw counter so the chart shows 'requests since boot' instead of a useful rate.
  • Using a gauge for request counts or a counter for free memory — the type must match the metric's behavior.
  • Adding a per-user or per-request label value, causing cardinality explosion and slow queries.
  • Setting scrape_interval too short (wasted CPU and storage) or too long (missing short spikes).
  • Computing histogram quantiles over the wrong window or ignoring bucket resets on restarts.
Root cause analysis

Common issues

Error
A metric disappears from charts after a few minutes
Root cause

The target stopped reporting, the exporter crashed, or the scrape started failing.

Fix

Query up for that job and check /api/v1/targets; restart the exporter and verify its /metrics endpoint with curl.

Error
Histogram quantile returns a wrong or empty value
Root cause

A summary was used instead of a histogram, or too few observations fall into the requested bucket.

Fix

Switch to histogram buckets with sufficient resolution around the target percentile and aggregate with sum by (le).

Error
Prometheus runs out of memory or disk
Root cause

Cardinality explosion or a retention setting larger than the capacity supports.

Fix

Audit label usage, raise --storage.tsdb.retention.time only after reducing cardinality, and add limits on series per metric.

When things break

Debugging tips

  • When a metric is missing, check /api/v1/targets first — a failing scrape is the most common cause.
  • Use up == 0 in queries to distinguish a dead target from a metric that was never created.
  • Test PromQL in the Expression browser before writing an alert rule so you can see the vector shape.
  • Watch for stale markers in the UI — a flat line dropping to nothing usually means the target restarted.
  • Run promtool check config and promtool check rules before any reload to catch YAML or syntax errors.
Interviews

Interview checkpoint

Practice

Hands-on lab

Run Prometheus, Scrape a Target and Query It
25 min · intermediate
In progress
Objectives
  • Write a minimal Prometheus configuration
  • Scrape a real node-exporter target
  • Verify targets and queries through the HTTP API
  • Stop the components when finished

Run Prometheus, Scrape a Target and Query It

25 min · intermediate

0/7 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which Prometheus metric type is a monotonically increasing total?

2

What does the up metric equal for a target when the scrape fails?

3

Summary quantiles can be aggregated across multiple instances.

Quick reference

Cheat sheet

PromQL essentials
rate(counter[5m])Per-second growth of a counter over 5 minutes
increase(counter[1h])Total increase over the window
sum by (instance) (...)Aggregate across one dimension
histogram_quantile(0.99, ...)Percentile from histogram buckets
predict_linear(metric[1h], 3600)Forecast a value one hour ahead
Operate Prometheus
curl /api/v1/targetsScrape target health
curl '/api/v1/query?query=up'Instant query via the HTTP API
promtool check configValidate prometheus.yml
promtool check rulesValidate recording/alerting rules
POST /-/reloadReload config without restart
Go deeper

Official documentation

Prometheus — Overview
docs
Prometheus — Metric Types
docs
PromQL Basics
docs
Histograms and Summaries — Best Practices
docs
Prometheus — Storage
docs