Storage
Advanced 12 min read Storage

Storage Performance and IO

Measuring and tuning the I/O path

IOPS, throughput, latency and burst — storage performance is measured at each layer. Matching IOPS/bandwidth to workload while avoiding throttling is how you keep databases fast and bills sane.

IOPSthroughputlatencyburstprovisioned IOPSswap
Prerequisites: Block Storage Volumes
Before we start

What you will learn

  • Decompose IOPS, throughput and latency
  • Provision IOPS where workloads need them
  • Benchmark and tune storage performance
In 60 seconds

IOPS = operations per second, throughput = bytes/sec (IOPS × block size), latency = per-op time. Volume types fix the IOPS/throughput envelope; DBs need IOPS, media pipelines need throughput, neither wants latency spikes.

Introduction

What is this?

IOPS vs throughput: a workload with 4KB random writes wants many IOPS; a video copy wants GB/s. Volume archetypes: gp3 (3000 baseline IOPS, 125 MB/s, burstable), io2 (provisioned up to 256K IOPS), st1 (throughput), Ultra Disk/pd-ssd/io2 for extreme IOPS. Latency also depends on host type, EBS-optimized instances, and filesystem mount options (noatime).

Motivation

Why do you need it?

  • Throttled volumes bottleneck whole apps at exactly the moment they need speed.
  • Right IOPS sizing avoids the 'fast tier, idle tier' waste.
  • Understanding burst mandates the credit-buoyancy of burstable volumes.
  • Common interview: 'the DB is slow — is it IOPS, cache or network?'
Think about it

Mental model

Imagine

Kitchen staff capacity

Think of

IOPS/throughput

IOPS is how many dishes a kitchen can plate per minute (many small plates); throughput is how many kilograms it can move (few gigantic pots). Either maxed out — tiny appetizers or a 300-person banquet — shows up differently.

Mechanics

How it works

  • Every op touches the storage network: NIC + hypervisor + storage backend → add host IOPS budget.
  • Volume types set envelope: gp3/flat baseline, io2/provisioned, st1 throughput-focused.
  • Burst balances replenish with workload; sustained load beyond baseline throttles to baseline.
  • Mount options (noatime, NOCOW for DB on xfs) cut metadata ops.
  • Filesystem block size vs application pattern shapes effective throughput.

The IO path.

App
Filesystem
Volume
EBS host budget
Storage backend
Hands-on

Examples

Measure before you choose

Measure before you choose
# quick fio on a mounted volume
sudo fio --name=randwrite --ioengine=libaio --rw=randwrite \
  --bs=4k --size=2G --numjobs=4 --iodepth=32 --time_based \
  --runtime=30 --group_reporting
# reads underscore what the volume type actually sustains
sudo fio --name=readtest --rw=read --bs=1M --numjobs=8 --iodepth=32 \
  --runtime=30 --group_reporting

Separate random-IOPS and sequential-throughput benchmarks reveal which dimension your volume type provides.

Real world

In production

  • A MySQL cluster uses io2 volumes with 30K IOPS and monitors Amazon EBS latency on p99.
  • A rendering farm puts source textures on throughput-optimized st1 volumes read sequentially.
  • Ultra Disk serves a financial ticker's extreme random IOPS with sub-ms latency.
Guidelines

Best practices

  • Benchmark with your actual workload pattern (4K random vs 1M sequential).
  • Pick IOPS to cover p99 write spikes, not just averages.
  • Enable EBS-optimized/accelerated networking for tight I/O curves.
  • Use noatime and align filesystems; set mount io stats for iostat visibility.
Watch out

Common mistakes

  • Buying the biggest IOPS and never benchmarking the actual need.
  • Ignoring host-level ceilings: an un-optimized instance cap slices IOPS anyway.
  • Interpreting average IOPS when the app needs sustained, not burst-then-throttle.
  • Not using batching (iodepth): single-stream IO never saturates NVMe volumes.
Root cause analysis

Common issues

Error
IOPS sit at exact baseline regardless of load
Root cause

Burst balances exhausted

Fix

Provision a higher-IOPS volume type or reduce op size/batching to lower request rate.

Error
Low throughput on a 'fast' volume
Root cause

Instance network bandwidth ceiling or single-stream

Fix

Scale instance network budget, increase dept/parallelism, or move to a throughput type.

When things break

Debugging tips

  • Run iostat -x: await/r_await show device latency vs total runtime.
  • Compare measured IOPS to volume baseline: hitting baseline with sustained load = throttling.
  • Check EBS VolumeQueueLength and burst balance metrics.
  • Network-bound? 'netstat'/bandwidth metric on the instance reveals the NIC ceiling before the volume.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/1
1

A customer copies 1 GiB files. Which storage dimension matters most?

Go deeper

Official documentation

AWS — EBS volume types
docs
Azure — Disk types
docs
fio — flexible I/O tester
docs