Lesson 5 of 7 in DevOps and delivery fundamentals
Feedback loops and small batches
A feedback loop is the time between making a change and learning what it did. A small batch is a change with few moving parts, so when something goes wrong the cause is easy to locate. Shortening the loop and shrinking the batch go together.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: CI, continuous delivery and continuous deployment
The problem this solves
When a release bundles weeks of work from many people, a failure has many suspects. Finding the cause means reading a large diff and guessing, and fixing it means either rolling back everything or patching under pressure.
Long feedback loops make it worse. If you learn that a change broke something days later, you have forgotten the details, and other changes have piled on top of it.
Small, frequent changes reverse this. A failed release has one or two suspects. A failing test appears minutes after the commit. Both CI and fast monitoring exist to keep the loop short.
How it works
- People and clients
- Control
- Runs code
- Data
A developer pushes a small change through the pipeline into production, monitoring reports what happened, and that result reaches the developer quickly.
- Developer to Pipeline: 1 push
- Pipeline to Production: 2 release
- Production to Monitoring: 3 signals
- Monitoring to Developer: 4 result
- Other
- Control
A bundle of many changes that fails leaves a hard search for the cause, while a single small change that fails points straight at the cause.
- Many changes to One big release: 1 bundle
- One big release to Failure: 2 ship
- Failure to Hard to find cause: 3 investigate
- One change to Small release: 1 ship alone
- Small release to Failure: 2 ship
- Failure to Obvious cause: 3 investigate
The first picture is the loop you want. The distance from step 1 to step 4 is the thing to shorten. Fast tests in the pipeline and good signals from production both cut it.
The second picture compares two ways to ship the same work. Top row: many changes go out together, something fails, and every one of them is a suspect. Bottom row: each change goes out alone, so the failure points at the last thing that changed.
Small does not mean unfinished. Each change should leave the main line working. Work that is not ready for users can be merged behind a feature flag, which is a setting that keeps the code switched off until you turn it on.
Try it yourself
You will build a history of eight small commits with a bug hidden in one of them, then let Git find it. Linux, macOS or WSL, in a throwaway repo.
- 1Create eight small commits, one of which adds a bug
mkdir -p ~/devops-lab/small-batches && cd ~/devops-lab/small-batches git init git config user.name "Learner" git config user.email "learner@example.com" for i in 1 2 3 4 5 6 7 8; do if [ "$i" = 5 ]; then echo "BUG" >> app.txt; else echo "feature $i" >> app.txt; fi git add app.txt git commit -q -m "add feature $i" done git log --oneline
Commit 5 writes the word BUG, and every later commit keeps it.
git log --onelinelists eight commits, newest first. - 2Define what good means as a command
! grep -q BUG app.txt; echo "exit=$?"
You should see
exit=1
grep -qsucceeds when it finds BUG, and the leading!flips that, so the check passes only when BUG is absent. The exit code is 1 now because the latest commit is bad. - 3Let git bisect search the history
git bisect start git bisect bad HEAD git bisect good HEAD~7 git bisect run sh -c '! grep -q BUG app.txt'
HEAD~7is the first commit, which you know is good. Git checks out commits in between and runs your check on each. It ends by printing the first bad commit, which should be the one with the messageadd feature 5. - 4Return to normal
git bisect reset
This puts your repo back on the branch you started from. Eight commits needed only about three checks because bisect halves the range each time. One commit holding all eight changes would give Git nothing to narrow down.
What breaks, and where to look
Merging a long-lived branch prints CONFLICT (content): Merge conflict in <file> and Automatic merge failed; fix conflicts and then commit the result.
- WHY IT HAPPENS
- The branch and the main line both changed the same lines over a long time, so Git cannot combine them without help.
- CHECK FIRST
- Run
git diff --stat main...your-branchto see how much changed. Merge main into your branch often, or split the work into pieces that can merge on their own.
People bundle many changes into each release because the pipeline takes too long to run often.
- WHY IT HAPPENS
- A slow feedback loop makes each run expensive, so people run it less, which makes batches bigger and failures harder to diagnose.
- CHECK FIRST
- Read the time spent per stage in the CI run. Run the quickest checks first and move slow ones later. Martin Fowler's CI article suggests aiming for a build of roughly ten minutes.
Frequent merging leaves half-finished features visible to users, or breaks the build.
- WHY IT HAPPENS
- Small batches were applied without a way to hide incomplete work, so unfinished code goes live as soon as it merges.
- CHECK FIRST
- Check whether the new code path is behind a feature flag or reachable only from code that is not yet wired in. Each commit should still build and pass tests.
When not to use this
- Some changes cannot be shrunk to nothing, such as a database migration or a large dependency upgrade. Split them into backwards-compatible steps where you can, and plan the rest as a deliberate, slower rollout.
- Tiny commits only help if each one builds and passes tests. A series of broken intermediate commits makes
git bisectgive wrong answers.
Check yourself
What is a feedback loop in delivery?
The time from making a change to learning what it did. Shorter loops mean problems are found while the change is still fresh.
Why is a failed small release easier to fix than a failed big one?
Fewer things changed, so there are fewer suspects, and rolling back loses less work.
How does a feature flag help with small batches?
It lets unfinished code merge and deploy while staying switched off, so changes stay small without exposing work in progress.
Keep going
Lesson 5 of 92 in the DevOps path