Performance Testing
Advanced 14 min read Performance Testing

Profiling & Bottleneck Analysis

Profiling turns 'the server is slow' into a ranked list of hotspots: CPU samples, flame graphs, memory and GC analysis, thread dumps and distributed traces. Learn to measure first, fix the constraint, and prove the improvement.

profilingflame graphpprofperfGCthread dumpbottleneck
Prerequisites: Load Models & Scenarios
Introduction

What is this?

Profiling and bottleneck analysis is the detective work after a load test: instead of guessing, you measure where CPU time, memory, locks, I/O and database time actually go under load. The toolchain spans CPU profilers (perf, pprof, async-profiler), heap and GC analysis, thread and lock dumps, and distributed tracing to correlate an endpoint's latency with its backend calls. The output is a ranked list of hotspots and a root-cause fix — not a vague 'the server is slow'.

Motivation

Why do you need it?

  • Latency and error-rate numbers tell you something is wrong; profiles tell you exactly what — the difference between symptoms and causes.
  • CPU hotspots, lock contention, GC pressure and I/O waits each need different fixes, and only measurement tells them apart.
  • A profile taken during the load test is worth a thousand after-the-fact guesses; transient bottlenecks vanish once load stops.
  • Bottleneck analysis turns capacity planning from intuition into a model: fix the constraint and capacity improves proportionally.
  • Distributed tracing ties a slow page to the specific downstream call, so teams stop firefighting the wrong service.
Think about it

Mental model

Imagine

A doctor running tests instead of guessing

Think of

Profiling a slow system

You don't treat 'fever' generically — you run blood work, cultures and imaging to find the infection. Profilers and traces are the equivalent diagnostics for a slow service.

Mechanics

How it works

  • Reproduce the load that triggers the symptom and keep it running while you collect samples.
  • Capture a CPU profile (perf record, async-profiler or pprof) and visualize it as a flame graph.
  • Snapshot memory and GC behavior: heap dumps, GC logs and allocation profiles reveal leaks and allocation churn.
  • Correlate with blocking: thread dumps and lock contention show stalls that CPU time cannot.
  • For distributed systems, follow a slow trace: span-by-span time reveals the downstream culprit.
  • Confirm the fix by re-running the same load test and comparing the profile and metrics to the baseline.

From a slow endpoint to a root-cause fix.

Load test
CPU profile
Flame graph
Memory / locks
Distributed trace
Fix and re-test
Reference

Key commands

perf record -F 99 -g -- ./my-service

Sample the process at 99 Hz and record call stacks for flame graph generation.

perf report

Interactive tree view of recorded samples, ranked by cost.

Overhead  Command  Shared Object  Symbol
12.4%   server   libc.so.6      memcpy
top -H -p <pid>

Watch per-thread CPU and identify the busiest thread or goroutine.

jcmd <pid> GC.heap_info

Dump JVM heap status and GC behavior without restarting the application.

jstack <pid> > threaddump.txt

Capture a Java thread dump to spot deadlocks and blocked threads.

go tool pprof -top http://localhost:6060/debug/pprof/profile

Profile a Go service over HTTP and show the top hotspots directly.

Hands-on

Examples

Profiling a Go service under load with pprof

Profiling a Go service under load with pprof
# Sample the CPU for 30 seconds while load is running
go tool pprof -seconds 30 http://localhost:6060/debug/pprof/profile
 
# Rank functions by self time
go tool pprof -top /tmp/pprof.pprof.go1.20.cpu.001.pb.gz
 
# Render a flame graph
go tool pprof -svg /tmp/pprof.pprof.go1.20.cpu.001.pb.gz > cpu.svg
 
# Blocking/contention profile (goroutine stalls)
go tool pprof http://localhost:6060/debug/pprof/block

pprof samples the running Go service's CPU for 30 seconds while the load test is in flight. -top ranks functions by self time to find the hotspot, and -svg renders a flame graph you can inspect visually. The block profile shows goroutine stalls that CPU time alone won't reveal.

Real world

In production

  • A flame graph from a staging load test turns a vague 'slow endpoint' into a named function call to optimize.
  • GC logs and heap profiles reveal allocation churn that causes throughput collapse under load.
  • Thread dumps taken at the moment of a latency spike catch locks and pool stalls that disappear when load stops.
  • Per-request tracing attributes latency to the actual downstream dependency before any code is touched.
Guidelines

Best practices

  • Profile while the load is still running — a post-mortem profile after traffic stops misses the bottleneck.
  • Collect CPU, memory and blocking data in parallel; the same symptom often has more than one contributing factor.
  • Compare profiles before and after a fix, not just 'is it faster now'.
  • Profile in an environment that matches production — same CPU, GC settings and data volume.
  • Reduce the search space with traces first: know which service and span is slow before deep-profiling one binary.
Watch out

Common mistakes

  • Profiling a single request in isolation and assuming the hotspot is the same under concurrency — locks only appear with load.
  • Reading the widest flame graph frame and 'fixing' it without checking self time versus cumulative time.
  • Optimizing CPU when the real constraint is I/O or connection-pool exhaustion — measured, not guessed.
  • Killing the load before capturing the profile, so the sampled workload doesn't match the symptom.
Root cause analysis

Common issues

Error
perf record failed to read the process / Operation not permitted
Root cause

perf needs elevated privileges or a permissive perf_event_paranoid setting.

Fix

Run with sudo, or set /proc/sys/kernel/perf_event_paranoid to -1 and verify the kernel config.

Error
profile has 0 samples
Root cause

The app exited, or sampling happened outside the slow window.

Fix

Keep the load running and use -seconds to capture a window inside it; verify the PID.

Error
flame graph is dominated by wait / blocked frames
Root cause

The bottleneck is contention or I/O, not CPU.

Fix

Collect lock and block profiles (pprof/block, jstack) and trace downstream calls instead.

When things break

Debugging tips

  • A flat CPU profile with high latency points to waiting (I/O, locks, pool) rather than computation.
  • Compare the profile during steady state versus during the spike — different hotspots mean different fixes.
  • Watch GC throughput (bytes/sec) alongside latency; heavy GC usually correlates with p99 cliffs.
  • Use the profile timestamps: only trust samples taken inside the observed slow window.
Interviews

Interview checkpoint

Practice

Hands-on lab

Profile a Bottleneck Under Load
40 min · advanced
In progress
Objectives
  • Run a k6 load test against a deliberately CPU-heavy Node service
  • Capture a CPU profile while the load is in flight
  • Read the flame chart to identify the hotspot
  • Apply a fix and prove the improvement with the identical workload

Profile a Bottleneck Under Load

40 min · advanced

0/5 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/3
1

Which tool samples a Go service's CPU over HTTP while it is under load?

2

A flame graph's widest frame at the top is always the code you should optimize.

3

Latency is high but CPU utilization is near zero. Which hypothesis is most likely?

Quick reference

Cheat sheet

Profiling toolbox
perf record -F 99 -g -- <cmd>CPU samples + call stacks
perf reportRanked hotspot tree
go tool pprof -seconds 30 :6060/debug/pprof/profileGo CPU profile over HTTP
jstack <pid>Java thread dump
jcmd <pid> GC.heap_infoJVM heap + GC state
top -H -p <pid>Per-thread CPU watch
Go deeper

Official documentation

perf wiki
docs
Brendan Gregg — Flame Graphs
docs
Go — net/http/pprof
docs
async-profiler
docs