Lesson 5 of 7 in DevOps and delivery fundamentals

Feedback loops and small batches

Core9 min

A feedback loop is the time between making a change and learning what it did. A small batch is a change with few moving parts, so when something goes wrong the cause is easy to locate. Shortening the loop and shrinking the batch go together.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: CI, continuous delivery and continuous deployment

The problem this solves

When a release bundles weeks of work from many people, a failure has many suspects. Finding the cause means reading a large diff and guessing, and fixing it means either rolling back everything or patching under pressure.

Long feedback loops make it worse. If you learn that a change broke something days later, you have forgotten the details, and other changes have piled on top of it.

Small, frequent changes reverse this. A failed release has one or two suspects. A failing test appears minutes after the commit. Both CI and fast monitoring exist to keep the loop short.

How it works

A short loop: change, release, observe, adjust
A short loop: change, release, observe, adjustA developer pushes a small change through the pipeline into production, monitoring reports what happened, and that result reaches the developer quickly.1 push2 release3 signals4 resultDevelopersmall changePipelinetests in minutesProductionMonitoringwhat happened
  • People and clients
  • Control
  • Runs code
  • Data

A developer pushes a small change through the pipeline into production, monitoring reports what happened, and that result reaches the developer quickly.

  1. Developer to Pipeline: 1 push
  2. Pipeline to Production: 2 release
  3. Production to Monitoring: 3 signals
  4. Monitoring to Developer: 4 result
Why small batches are easier to debug
Why small batches are easier to debugA bundle of many changes that fails leaves a hard search for the cause, while a single small change that fails points straight at the cause.1 bundle2 ship3investigate1 ship alone2 ship3investigateMany changesOne big releaseFailureHard to find causemany suspectsOne changeSmall releaseFailureObvious causeone suspect
  • Other
  • Control

A bundle of many changes that fails leaves a hard search for the cause, while a single small change that fails points straight at the cause.

  1. Many changes to One big release: 1 bundle
  2. One big release to Failure: 2 ship
  3. Failure to Hard to find cause: 3 investigate
  4. One change to Small release: 1 ship alone
  5. Small release to Failure: 2 ship
  6. Failure to Obvious cause: 3 investigate

The first picture is the loop you want. The distance from step 1 to step 4 is the thing to shorten. Fast tests in the pipeline and good signals from production both cut it.

The second picture compares two ways to ship the same work. Top row: many changes go out together, something fails, and every one of them is a suspect. Bottom row: each change goes out alone, so the failure points at the last thing that changed.

Small does not mean unfinished. Each change should leave the main line working. Work that is not ready for users can be merged behind a feature flag, which is a setting that keeps the code switched off until you turn it on.

Try it yourself

You will build a history of eight small commits with a bug hidden in one of them, then let Git find it. Linux, macOS or WSL, in a throwaway repo.

  1. 1
    Create eight small commits, one of which adds a bug
    mkdir -p ~/devops-lab/small-batches && cd ~/devops-lab/small-batches
    git init
    git config user.name "Learner"
    git config user.email "learner@example.com"
    for i in 1 2 3 4 5 6 7 8; do
      if [ "$i" = 5 ]; then echo "BUG" >> app.txt; else echo "feature $i" >> app.txt; fi
      git add app.txt
      git commit -q -m "add feature $i"
    done
    git log --oneline

    Commit 5 writes the word BUG, and every later commit keeps it. git log --oneline lists eight commits, newest first.

  2. 2
    Define what good means as a command
    ! grep -q BUG app.txt; echo "exit=$?"

    You should see

    exit=1

    grep -q succeeds when it finds BUG, and the leading ! flips that, so the check passes only when BUG is absent. The exit code is 1 now because the latest commit is bad.

  3. 3
    Let git bisect search the history
    git bisect start
    git bisect bad HEAD
    git bisect good HEAD~7
    git bisect run sh -c '! grep -q BUG app.txt'

    HEAD~7 is the first commit, which you know is good. Git checks out commits in between and runs your check on each. It ends by printing the first bad commit, which should be the one with the message add feature 5.

  4. 4
    Return to normal
    git bisect reset

    This puts your repo back on the branch you started from. Eight commits needed only about three checks because bisect halves the range each time. One commit holding all eight changes would give Git nothing to narrow down.

What breaks, and where to look

  • Merging a long-lived branch prints CONFLICT (content): Merge conflict in <file> and Automatic merge failed; fix conflicts and then commit the result.

    WHY IT HAPPENS
    The branch and the main line both changed the same lines over a long time, so Git cannot combine them without help.
    CHECK FIRST
    Run git diff --stat main...your-branch to see how much changed. Merge main into your branch often, or split the work into pieces that can merge on their own.
  • People bundle many changes into each release because the pipeline takes too long to run often.

    WHY IT HAPPENS
    A slow feedback loop makes each run expensive, so people run it less, which makes batches bigger and failures harder to diagnose.
    CHECK FIRST
    Read the time spent per stage in the CI run. Run the quickest checks first and move slow ones later. Martin Fowler's CI article suggests aiming for a build of roughly ten minutes.
  • Frequent merging leaves half-finished features visible to users, or breaks the build.

    WHY IT HAPPENS
    Small batches were applied without a way to hide incomplete work, so unfinished code goes live as soon as it merges.
    CHECK FIRST
    Check whether the new code path is behind a feature flag or reachable only from code that is not yet wired in. Each commit should still build and pass tests.

When not to use this

  • Some changes cannot be shrunk to nothing, such as a database migration or a large dependency upgrade. Split them into backwards-compatible steps where you can, and plan the rest as a deliberate, slower rollout.
  • Tiny commits only help if each one builds and passes tests. A series of broken intermediate commits makes git bisect give wrong answers.

Check yourself

What is a feedback loop in delivery?

The time from making a change to learning what it did. Shorter loops mean problems are found while the change is still fresh.

Why is a failed small release easier to fix than a failed big one?

Fewer things changed, so there are fewer suspects, and rolling back loses less work.

How does a feature flag help with small batches?

It lets unfinished code merge and deploy while staying switched off, so changes stay small without exposing work in progress.

Lesson 5 of 92 in the DevOps path

Search

Search cheat sheets and projects