Lesson 6 of 7 in DevOps and delivery fundamentals
DORA metrics: measuring delivery health
DORA metrics are a small set of measures from the DevOps Research and Assessment programme that describe how a team delivers software. Some measure throughput, how fast changes reach production, and some measure instability, how often those changes cause trouble.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Feedback loops and small batches
The problem this solves
Teams that want to improve delivery need a way to tell whether they are. Opinions such as feeling fast or feeling safe are easy to argue with and easy to fool yourself with.
DORA names a handful of measures that can be taken from data a team already has: version control, the pipeline, and incident records. They give two views at once, speed and stability, so improving one at the cost of the other shows up.
The metrics are descriptions of a system, not goals to push. Used well they point to where to look. Used as targets they get gamed.
How it works
- People and clients
- Control
- Other
Lead time runs from commit to production deploy, the share of deploys needing a fix is the failure rate, the time to recover from a failed deploy is recovery time, and the gap between deploys gives frequency.
- Commit to Deploy: 1 lead time
- Deploy to Failed deploy: 2 some need a fix
- Failed deploy to Recovered: 3 recovery time
- Deploy to Next deploy: 4 deploy frequency
Start at the left. Change lead time is how long a change takes to go from committed in version control to running in production. Deployment frequency is how often you deploy, or the time between deployments, shown by the dashed arrow to the next deploy.
Recovery time is how long it takes to recover from a deployment that failed and needed immediate intervention. DORA groups it with lead time and frequency as throughput. The instability side starts with change failure rate, the share of deployments that need immediate intervention, such as a rollback or a hotfix.
DORA's current guide calls recovery time failed deployment recovery time. The older four-key lists called it time to restore. The guide also adds a fifth measure, deployment rework rate, the share of deployments that are unplanned and happen because of a production incident. Check dora.dev for the wording in force when you read this.
Try it yourself
This uses a small practice file with invented numbers, only to show how the arithmetic works. Linux, macOS or WSL. Real data would come from your pipeline and incident tracker.
- 1Create a practice log of six deployments
mkdir -p ~/devops-lab/dora && cd ~/devops-lab/dora cat > deploys.csv <<'EOF' day,lead_hours,failed,restore_minutes 1,4,no,0 1,6,no,0 2,3,yes,45 3,5,no,0 3,2,no,0 5,8,yes,30 EOF
Each row is one deployment: the day it happened, hours from commit to deploy, whether it needed immediate intervention, and minutes to recover.
- 2Change failure rate
awk -F, 'NR>1 {n++; if ($3=="yes") f++} END {printf "deploys=%d failed=%d change_fail_rate=%.0f%%\n", n, f, 100*f/n}' deploys.csvYou should see
deploys=6 failed=2 change_fail_rate=33%
- 3Change lead time
awk -F, 'NR>1 {s+=$2; n++} END {printf "mean lead time = %.1f hours\n", s/n}' deploys.csvYou should see
mean lead time = 4.7 hours
A mean hides outliers. With real data, many teams look at the median or a high percentile as well.
- 4Recovery time for the failed deployments
awk -F, 'NR>1 && $3=="yes" {s+=$4; n++} END {printf "mean recovery = %.1f minutes\n", s/n}' deploys.csvYou should see
mean recovery = 37.5 minutes
- 5Deployment frequency
awk -F, 'NR>1 {d[$1]=1; n++} END {for (k in d) days++; printf "%d deploys on %d different days\n", n, days}' deploys.csvYou should see
6 deploys on 4 different days
Frequency needs a time window, such as per day or per week. Pick one and keep it the same when you compare periods.
What breaks, and where to look
Change failure rate reads 0 percent although people remember several bad releases.
- WHY IT HAPPENS
- Nothing defines or records a failed deployment, so no row is ever marked as failed.
- CHECK FIRST
- Agree on a definition, for example any deployment followed by a rollback, hotfix or incident ticket, and count those in the deploy history for a past month.
Lead time shows days, yet developers say code is merged within hours.
- WHY IT HAPPENS
- The clock starts at ticket creation, or ends at a release train that runs weekly, instead of running from commit to production deploy.
- CHECK FIRST
- Look at the two timestamps the number uses. DORA's definition runs from the commit to version control to the deploy to production.
The numbers improve while the product or the on-call experience gets worse.
- WHY IT HAPPENS
- A metric became a target, so people optimise the number, for example by deploying trivial changes to raise frequency.
- CHECK FIRST
- Read throughput and instability together, and check them against something users feel, such as errors or support requests. Ask who uses the numbers and for what.
When not to use this
- DORA metrics describe how software is delivered. They say nothing about whether you are building the right thing for users.
- With only a handful of deployments in a period, a rate swings wildly, since one failure changes it a lot. Compare longer windows before drawing conclusions.
- Do not use them to rank teams working on different products, because context changes what a good number means.
Check yourself
Which DORA metrics describe throughput and which describe instability?
Throughput: change lead time, deployment frequency and recovery time. Instability: change failure rate and, in the current guide, deployment rework rate.
Where does change lead time start and end?
It starts when the change is committed to version control and ends when it is deployed to production.
Why track speed and stability together?
Speed alone can hide a rise in failures, and stability alone can hide that you ship rarely. Together they show whether you are improving without trading one for the other.
Lesson 6 of 92 in the DevOps path