Performance Testing
Beginner 12 min read Performance Testing

Metrics & Baselines

A baseline is a trusted reference point: the latency distribution, throughput and error rate of the current version under a known workload. Metrics are the measurements you take on every run — learn to capture percentiles correctly and compare them fairly.

baselinepercentilep95p99latency distributionthroughput
Prerequisites: Types of Performance Testing
Introduction

What is this?

Baselines are the trusted reference points your performance work is measured against: the response-time distribution, throughput and resource usage of the current version under a known workload. Metrics are the actual measurements — percentiles, error rates, requests per second and utilization — captured on every run. Together they let you answer 'is this version faster or slower than last week' with numbers instead of opinions.

Motivation

Why do you need it?

  • Without a baseline every test result floats: you cannot say whether p95 of 300ms is good or bad.
  • Percentiles matter more than averages — a single slow request can inflate the mean while p99 tells the real story for the tail.
  • Baselines make performance a tracked quantity: you can alert, gate releases and prove improvement over time.
  • A stored baseline lets you compare across environments and versions even when infrastructure changes.
  • Metric deltas — client latency vs server CPU vs DB time — pinpoint where a regression actually lives.
Think about it

Mental model

Imagine

The bathroom scale vs your health history

Think of

Raw metrics vs a baseline

A single weigh-in tells you little; tracking weight weekly and comparing to your history is what shows trends. Baselines give raw performance metrics their context.

Mechanics

How it works

  • Run a load test against a known version and workload, and record the full distribution: p50, p90, p95, p99, max, plus throughput and error rate.
  • Store those numbers as the baseline, tagged with app version, environment and test profile so later comparisons are meaningful.
  • Repeat the same workload on a new build or environment and collect the same set of metrics.
  • Compare distributions, not averages — for example, p99 within 10% of baseline means pass; beyond that, investigate.
  • Review deltas in server-side metrics (CPU, GC, DB time) to explain any client-side change.

From raw measurements to an actionable regression comparison.

Known workload
Capture metrics
Store baseline
Re-run workload
Compare distributions
Approve or investigate
Reference

Key commands

k6 run --summary-trend-stats 'avg,p(50),p(90),p(95),p(99)' script.js

Pin the exact percentiles printed in the summary so runs are comparable.

http_req_duration: avg=45ms p(50)=42ms p(90)=58ms p(95)=63ms p(99)=90ms
k6 run --out json=results.json script.js

Stream every metric point to JSON for post-processing and long-term storage.

ab -n 2000 -c 50 http://api.example.com/v1/items

ApacheBench gives requests/sec plus the p50/p66/p75/p98/p99/p100 latency table directly.

Requests per second: 812.4 [#/sec] (mean)
wrk -t4 -c100 -d30s http://api.example.com/v1/items

Measure the latency distribution and throughput at high concurrency with wrk.

Latency Distribution: 50% 12.34ms, 90% 18.45ms, 99% 25.67ms
curl -s -o /dev/null -w 'code=%{http_code} ttfb=%{time_starttransfer}' https://api.example.com/

Single-request probe to split time to first byte from total time.

code=200 ttfb=0.142
Hands-on

Examples

Extracting baseline percentiles from a k6 JSON result stream

Extracting baseline percentiles from a k6 JSON result stream
k6 run --out json=results.json script.js
 
jq -s '
  [.[] | select(.metric == "http_req_duration" and .type == "Point") | .data.value]
  | sort
  | { count: length,
      p50: .[(length * 0.5 | floor)],
      p95: .[(length * 0.95 | floor)],
      p99: .[(length * 0.99 | floor)] }
' results.json

k6 streams every metric point to results.json with --out json. jq filters only http_req_duration samples, sorts them, and computes p50, p95 and p99. The result is a small, machine-readable baseline you can store and diff against the next run.

Real world

In production

  • Performance budgets in CI compare the new build's p95 to the stored baseline and fail the build over the threshold.
  • Baselines tagged with app version and infra config make capacity reviews compare like-for-like.
  • Historical baseline charts feed the ops dashboard, alerting on p99 drift before users complain.
  • Release notes ship with before/after latency numbers proven against the same baseline workload.
Guidelines

Best practices

  • Record the full distribution, not just the average — include p50, p90, p95, p99 and max.
  • Pin the baseline to a specific version, environment and test profile so comparisons are apples-to-apples.
  • Capture server-side metrics (CPU, memory, GC, slow queries) at the same time as client latency.
  • Re-baseline deliberately after major infrastructure changes; an old baseline becomes the wrong yardstick.
  • Automate baseline storage in a file or database with a schema that includes the timestamp and git SHA.
Watch out

Common mistakes

  • Gating releases on the mean, which hides a heavy tail of slow requests.
  • Comparing runs with different workload profiles or environment sizes and treating them as equivalent.
  • Forgetting to warm up before capturing the baseline, so caches skew the numbers.
  • Storing baselines as prose in a doc nobody updates instead of machine-readable data.
Root cause analysis

Common issues

Error
http_req_duration p99 is 10x the baseline
Root cause

A regression in a hot code path or a resource that only degrades under load.

Fix

Profile the endpoint during the test, correlate with server CPU and GC, and bisect commits since the baseline.

Error
Request rate is much lower than baseline at the same concurrency
Root cause

Client-side limits (file descriptors, HTTP/1.1 connection reuse) or server throttling.

Fix

Check the generator's socket limits and the server's rate-limiting and connection-queue settings.

Error
Baseline captured but the new run differs because the dataset changed
Root cause

Test data volume or shape changed between runs.

Fix

Freeze a seed dataset, or record the dataset size and cardinality in the baseline metadata.

When things break

Debugging tips

  • Plot latency percentiles over time; a p99 spike with a flat p50 points to tail latency in a specific code path.
  • Split the waterfall: DNS, connect, TLS, TTFB and transfer times isolate where the latency lives.
  • Compare the same workload across environments — if staging p95 matches prod, the app is fine and the infra is the variable.
  • Check error-rate metrics separately; retries and timeouts inflate apparent latency and throughput.
Interviews

Interview checkpoint

Practice

Hands-on lab

Capture and Compare a Baseline
25 min · beginner
In progress
Objectives
  • Run a fixed workload against a target endpoint
  • Record latency percentiles and throughput as a baseline
  • Introduce a server-side change and re-run the identical workload
  • Compare the distributions and quantify the difference

Capture and Compare a Baseline

25 min · beginner

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which metric best represents the experience of your slowest typical users?

2

A baseline is only meaningful when compared against runs with the same workload and environment.

3

Over a 2-hour soak test, p99 climbs from 80ms to 220ms while p50 stays flat. What does this most likely indicate?

Quick reference

Cheat sheet

Percentiles and baseline vocabulary
p50 / medianTypical user experience
p95Common SLO target
p99Tail latency, user-perceived jank
maxWorst outlier — often noise
requests/secThroughput, the capacity axis
error rate %Failed requests vs total
Go deeper

Official documentation

k6 — Metrics
docs
Prometheus — Histograms
docs
Google SRE Workbook — Implementing SLOs
docs
ApacheBench — ab manual
docs