Lesson 7 of 8 in CI/CD pipelines

Deployment strategies

Advanced8 min

A deployment strategy is the plan for moving users from the old version of an application to the new one. The main options are rolling, blue-green and canary, and they trade speed, cost and risk differently.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Environments and approvals, Containers with Docker

The problem this solves

Replacing the running version in one step means downtime while the new one starts, and a bad release hits every user at once. If it is broken, you are scrambling to rebuild the old version while people are affected.

A strategy limits the damage. It decides how many users see the new version, for how long both versions exist, and how quickly you can return to the old one.

How it works

Traffic is split between the old and new version
Traffic is split between the old and new versionUsers reach a load balancer, which sends most requests to the old version and a small share to the new version, and the pipeline changes that share step by step.1 request2 most traffic3 small share4 widen the shareUserssend requestsLoad balancerchooses a versionOld versionv1New versionv2Pipelinechanges the split
  • People and clients
  • Network
  • Runs code
  • Control

Users reach a load balancer, which sends most requests to the old version and a small share to the new version, and the pipeline changes that share step by step.

  1. Users to Load balancer: 1 request
  2. Load balancer to Old version: 2 most traffic
  3. Load balancer to New version: 3 small share
  4. Pipeline to Load balancer: 4 widen the share

Rolling replaces instances a few at a time. The old and new versions run side by side for a while, and no extra copy of the whole system is needed. Rolling back means rolling the old version out again, which takes about as long as the first rollout.

Blue-green keeps two complete environments. Blue serves users while you deploy and test the new version on green. Then you switch the router to green in one step. Rollback is switching back, which is fast, but you pay for two environments and any shared database must work with both versions.

Canary sends a small share of real traffic, say a few percent, to the new version first. You watch error rates and latency for the new version and only widen the share if they look healthy. It catches problems that tests miss, but it needs per-version metrics and a router that can split traffic.

All three run two versions at the same time for some period, so database changes must be compatible with both. A common approach is to add new columns first, deploy the code, and remove old columns in a later release.

Try it yourself

No real servers are needed. A symlink named current plays the router, and two folders play the blue and green environments. Run this in a throwaway folder on Linux or macOS.

  1. 1
    Create blue and green releases and point current at blue
    mkdir -p ~/bg-demo/releases/blue ~/bg-demo/releases/green && cd ~/bg-demo
    echo "version 1" > releases/blue/index.txt
    echo "version 2" > releases/green/index.txt
    ln -sfn releases/blue current
    cat current/index.txt

    You should see

    version 1

    Users only ever read through current. Blue is live. Green holds the new release and can be tested directly at releases/green without any user seeing it.

  2. 2
    Switch to green
    ln -sfn releases/green current
    cat current/index.txt

    You should see

    version 2

    One link change moves all users to the new version. A truly atomic swap would create a new link and rename it over the old one with mv -T, because ln -sfn briefly removes the old link.

  3. 3
    Roll back
    ln -sfn releases/blue current
    cat current/index.txt

    You should see

    version 1

    Rollback is the same switch in the other direction, which is the main benefit of blue-green. Blue was never touched, so it is still exactly as it was.

  4. 4
    Simulate a canary split
    new=0
    for i in $(seq 1 100); do
      [ $((RANDOM % 10)) -eq 0 ] && new=$((new + 1))
    done
    echo "requests to the new version: $new of 100"

    This is bash. About one request in ten goes to the new version, so the number varies around ten each time. Real canaries do this in a load balancer or service mesh, not in a loop. Clean up with rm -rf ~/bg-demo.

What breaks, and where to look

  • After the switch, the old version starts throwing errors such as a missing column, or the new version fails on old data.

    WHY IT HAPPENS
    Both versions share one database, and the schema change was not compatible with both. During a rolling or canary release, both versions are live at once.
    CHECK FIRST
    Look at the database change in the release. Split it into an additive step first and a cleanup step in a later release, and test the old code against the new schema.
  • The canary stage was green, yet users reported a problem.

    WHY IT HAPPENS
    The check looked at overall metrics, where the small canary share was drowned out, or nobody watched at all.
    CHECK FIRST
    Chart error rate and latency per version, not in total, and decide your success threshold before starting the rollout.
  • The switch to green works, but green behaves differently from blue.

    WHY IT HAPPENS
    The idle environment drifted. It was built from a different image, config or secret set than the live one.
    CHECK FIRST
    Build both environments from the same artifact and the same configuration source, and run smoke tests against green before switching.

When not to use this

  • Blue-green doubles your infrastructure cost while both exist. For a small app with a few seconds of acceptable downtime, a simple restart or a rolling update is enough.
  • Canary needs good monitoring. Without per-version metrics it only delays the full rollout and tells you nothing.
  • A rollback cannot undo data that the new version already wrote.

Check yourself

Why is rollback quick in blue-green?

The old environment is still running unchanged, so you only switch the router back.

What must you watch during a canary release?

Error rate and latency for the new version specifically, compared with the old version.

Why do database changes need extra care in all three strategies?

Old and new versions run at the same time against the same data, so the schema must work for both.

Lesson 53 of 92 in the DevOps path

Search

Search cheat sheets and projects