Lesson 6 of 7 in DevOps and delivery fundamentals

DORA metrics: measuring delivery health

Advanced10 min

DORA metrics are a small set of measures from the DevOps Research and Assessment programme that describe how a team delivers software. Some measure throughput, how fast changes reach production, and some measure instability, how often those changes cause trouble.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Feedback loops and small batches

The problem this solves

Teams that want to improve delivery need a way to tell whether they are. Opinions such as feeling fast or feeling safe are easy to argue with and easy to fool yourself with.

DORA names a handful of measures that can be taken from data a team already has: version control, the pipeline, and incident records. They give two views at once, speed and stability, so improving one at the cost of the other shows up.

The metrics are descriptions of a system, not goals to push. Used well they point to where to look. Used as targets they get gamed.

How it works

Where each DORA metric is measured on the timeline
Where each DORA metric is measured on the timelineLead time runs from commit to production deploy, the share of deploys needing a fix is the failure rate, the time to recover from a failed deploy is recovery time, and the gap between deploys gives frequency.1 lead time2 some needa fix3 recoverytime4 deploy frequencyCommitin version controlDeployto productionFailed deployneeds immediate fixRecoveredservice healthy againNext deploy
  • People and clients
  • Control
  • Other

Lead time runs from commit to production deploy, the share of deploys needing a fix is the failure rate, the time to recover from a failed deploy is recovery time, and the gap between deploys gives frequency.

  1. Commit to Deploy: 1 lead time
  2. Deploy to Failed deploy: 2 some need a fix
  3. Failed deploy to Recovered: 3 recovery time
  4. Deploy to Next deploy: 4 deploy frequency

Start at the left. Change lead time is how long a change takes to go from committed in version control to running in production. Deployment frequency is how often you deploy, or the time between deployments, shown by the dashed arrow to the next deploy.

Recovery time is how long it takes to recover from a deployment that failed and needed immediate intervention. DORA groups it with lead time and frequency as throughput. The instability side starts with change failure rate, the share of deployments that need immediate intervention, such as a rollback or a hotfix.

DORA's current guide calls recovery time failed deployment recovery time. The older four-key lists called it time to restore. The guide also adds a fifth measure, deployment rework rate, the share of deployments that are unplanned and happen because of a production incident. Check dora.dev for the wording in force when you read this.

Try it yourself

This uses a small practice file with invented numbers, only to show how the arithmetic works. Linux, macOS or WSL. Real data would come from your pipeline and incident tracker.

  1. 1
    Create a practice log of six deployments
    mkdir -p ~/devops-lab/dora && cd ~/devops-lab/dora
    cat > deploys.csv <<'EOF'
    day,lead_hours,failed,restore_minutes
    1,4,no,0
    1,6,no,0
    2,3,yes,45
    3,5,no,0
    3,2,no,0
    5,8,yes,30
    EOF

    Each row is one deployment: the day it happened, hours from commit to deploy, whether it needed immediate intervention, and minutes to recover.

  2. 2
    Change failure rate
    awk -F, 'NR>1 {n++; if ($3=="yes") f++} END {printf "deploys=%d failed=%d change_fail_rate=%.0f%%\n", n, f, 100*f/n}' deploys.csv

    You should see

    deploys=6 failed=2 change_fail_rate=33%
  3. 3
    Change lead time
    awk -F, 'NR>1 {s+=$2; n++} END {printf "mean lead time = %.1f hours\n", s/n}' deploys.csv

    You should see

    mean lead time = 4.7 hours

    A mean hides outliers. With real data, many teams look at the median or a high percentile as well.

  4. 4
    Recovery time for the failed deployments
    awk -F, 'NR>1 && $3=="yes" {s+=$4; n++} END {printf "mean recovery = %.1f minutes\n", s/n}' deploys.csv

    You should see

    mean recovery = 37.5 minutes
  5. 5
    Deployment frequency
    awk -F, 'NR>1 {d[$1]=1; n++} END {for (k in d) days++; printf "%d deploys on %d different days\n", n, days}' deploys.csv

    You should see

    6 deploys on 4 different days

    Frequency needs a time window, such as per day or per week. Pick one and keep it the same when you compare periods.

What breaks, and where to look

  • Change failure rate reads 0 percent although people remember several bad releases.

    WHY IT HAPPENS
    Nothing defines or records a failed deployment, so no row is ever marked as failed.
    CHECK FIRST
    Agree on a definition, for example any deployment followed by a rollback, hotfix or incident ticket, and count those in the deploy history for a past month.
  • Lead time shows days, yet developers say code is merged within hours.

    WHY IT HAPPENS
    The clock starts at ticket creation, or ends at a release train that runs weekly, instead of running from commit to production deploy.
    CHECK FIRST
    Look at the two timestamps the number uses. DORA's definition runs from the commit to version control to the deploy to production.
  • The numbers improve while the product or the on-call experience gets worse.

    WHY IT HAPPENS
    A metric became a target, so people optimise the number, for example by deploying trivial changes to raise frequency.
    CHECK FIRST
    Read throughput and instability together, and check them against something users feel, such as errors or support requests. Ask who uses the numbers and for what.

When not to use this

  • DORA metrics describe how software is delivered. They say nothing about whether you are building the right thing for users.
  • With only a handful of deployments in a period, a rate swings wildly, since one failure changes it a lot. Compare longer windows before drawing conclusions.
  • Do not use them to rank teams working on different products, because context changes what a good number means.

Check yourself

Which DORA metrics describe throughput and which describe instability?

Throughput: change lead time, deployment frequency and recovery time. Instability: change failure rate and, in the current guide, deployment rework rate.

Where does change lead time start and end?

It starts when the change is committed to version control and ends when it is deployed to production.

Why track speed and stability together?

Speed alone can hide a rise in failures, and stability alone can hide that you ship rarely. Together they show whether you are improving without trading one for the other.

Lesson 6 of 92 in the DevOps path

Search

Search cheat sheets and projects