Lesson 7 of 8 in CI/CD pipelines
Deployment strategies
A deployment strategy is the plan for moving users from the old version of an application to the new one. The main options are rolling, blue-green and canary, and they trade speed, cost and risk differently.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Environments and approvals, Containers with Docker
The problem this solves
Replacing the running version in one step means downtime while the new one starts, and a bad release hits every user at once. If it is broken, you are scrambling to rebuild the old version while people are affected.
A strategy limits the damage. It decides how many users see the new version, for how long both versions exist, and how quickly you can return to the old one.
How it works
- People and clients
- Network
- Runs code
- Control
Users reach a load balancer, which sends most requests to the old version and a small share to the new version, and the pipeline changes that share step by step.
- Users to Load balancer: 1 request
- Load balancer to Old version: 2 most traffic
- Load balancer to New version: 3 small share
- Pipeline to Load balancer: 4 widen the share
Rolling replaces instances a few at a time. The old and new versions run side by side for a while, and no extra copy of the whole system is needed. Rolling back means rolling the old version out again, which takes about as long as the first rollout.
Blue-green keeps two complete environments. Blue serves users while you deploy and test the new version on green. Then you switch the router to green in one step. Rollback is switching back, which is fast, but you pay for two environments and any shared database must work with both versions.
Canary sends a small share of real traffic, say a few percent, to the new version first. You watch error rates and latency for the new version and only widen the share if they look healthy. It catches problems that tests miss, but it needs per-version metrics and a router that can split traffic.
All three run two versions at the same time for some period, so database changes must be compatible with both. A common approach is to add new columns first, deploy the code, and remove old columns in a later release.
Try it yourself
No real servers are needed. A symlink named current plays the router, and two folders play the blue and green environments. Run this in a throwaway folder on Linux or macOS.
- 1Create blue and green releases and point current at blue
mkdir -p ~/bg-demo/releases/blue ~/bg-demo/releases/green && cd ~/bg-demo echo "version 1" > releases/blue/index.txt echo "version 2" > releases/green/index.txt ln -sfn releases/blue current cat current/index.txt
You should see
version 1
Users only ever read through
current. Blue is live. Green holds the new release and can be tested directly atreleases/greenwithout any user seeing it. - 2Switch to green
ln -sfn releases/green current cat current/index.txt
You should see
version 2
One link change moves all users to the new version. A truly atomic swap would create a new link and rename it over the old one with
mv -T, becauseln -sfnbriefly removes the old link. - 3Roll back
ln -sfn releases/blue current cat current/index.txt
You should see
version 1
Rollback is the same switch in the other direction, which is the main benefit of blue-green. Blue was never touched, so it is still exactly as it was.
- 4Simulate a canary split
new=0 for i in $(seq 1 100); do [ $((RANDOM % 10)) -eq 0 ] && new=$((new + 1)) done echo "requests to the new version: $new of 100"
This is bash. About one request in ten goes to the new version, so the number varies around ten each time. Real canaries do this in a load balancer or service mesh, not in a loop. Clean up with
rm -rf ~/bg-demo.
What breaks, and where to look
After the switch, the old version starts throwing errors such as a missing column, or the new version fails on old data.
- WHY IT HAPPENS
- Both versions share one database, and the schema change was not compatible with both. During a rolling or canary release, both versions are live at once.
- CHECK FIRST
- Look at the database change in the release. Split it into an additive step first and a cleanup step in a later release, and test the old code against the new schema.
The canary stage was green, yet users reported a problem.
- WHY IT HAPPENS
- The check looked at overall metrics, where the small canary share was drowned out, or nobody watched at all.
- CHECK FIRST
- Chart error rate and latency per version, not in total, and decide your success threshold before starting the rollout.
The switch to green works, but green behaves differently from blue.
- WHY IT HAPPENS
- The idle environment drifted. It was built from a different image, config or secret set than the live one.
- CHECK FIRST
- Build both environments from the same artifact and the same configuration source, and run smoke tests against green before switching.
When not to use this
- Blue-green doubles your infrastructure cost while both exist. For a small app with a few seconds of acceptable downtime, a simple restart or a rolling update is enough.
- Canary needs good monitoring. Without per-version metrics it only delays the full rollout and tells you nothing.
- A rollback cannot undo data that the new version already wrote.
Check yourself
Why is rollback quick in blue-green?
The old environment is still running unchanged, so you only switch the router back.
What must you watch during a canary release?
Error rate and latency for the new version specifically, compared with the old version.
Why do database changes need extra care in all three strategies?
Old and new versions run at the same time against the same data, so the schema must work for both.
Keep going
Lesson 53 of 92 in the DevOps path