High Availability
Intermediate 14 min read High Availability

Immutable & Zero-Downtime Deployments

Your deploys can be your best engineers

Deploys shouldn't be saboteurs: immutable images + rolling, blue/green and canary strategies give zero downtime, quick revert, and staged blast radii. The point: ship often without the Sunday outage prayer.

immutablecanaryblue greenrollingzero downtimerollbackdeployment strategy
Prerequisites: Containers & Kubernetes HA Fundamentals
Before we start

What you will learn

  • Compare rolling, blue/green, and canary strategies
  • Define readiness/health gates for safe progression
  • Plan rollback/revert paths deliberately
In 60 seconds

Immutable = every release a new image, never mutate a running instance. Strategies: rolling (staggered across replicas), blue/green (two full environments, flip traffic), canary (2-5% → graduated). Each has different detection, blast radius and cost; the safest playbook starts canary + auto-rollback on metric flop.

Introduction

What is this?

Technologies: Kubernetes Deployments (rolling strategy, maxSurge/maxUnavailable), Argo Rollouts (advanced rollout: canary, blue-green, analysis steps), ECS rolling/blue-green via CodeDeploy, Azure App Service slots, GCP Cloud Run revisions (traffic split), Lambda versioned aliases (weighted routing, gradual 1%). Rollout signals: readiness probe + liveness during rollout, poststart hooks, dashboards takeover detects. Gates: syntax/migration check, smoke/health, SLO windows (e.g., new canary must keep 3 min of good p95). Canary analysis: Argo AnalysisTemplates/Traffic managers prometheus queries; ECS/CodeDeploy equivalent. Rollback: previous deployment (kubectl rollout undo), re-pin image + scale canary, artifact-driven revert (redeploy good image). DB-migration discipline: backward-compatible migrations so old+new images both work during the shift.

Motivation

Why do you need it?

  • Zero-downtime releases deliver value twice a day without the 2am firewatch.
  • Canary broadcast catches the 3% regression before it sweeps the fleet.
  • Immutable images make rollback 'point at the old image' instead of untangling state.
  • Blast radius control is a calm-on-call superpower: overlap window shrinks to the wire.
Think about it

Mental model

Imagine

A bridge with two spans: you close one lane entirely, replace its bolts, then open and test the other under 1% of traffic

Think of

Blue/green deployment

You never bolt edges live; the new span (green) gets the load shift only after it proves out under trial traffic — and red lives where revision B lived until you say otherwise.

Mechanics

How it works

  • Build+push immutable image per commit; tag with sha.
  • Rollout strategy says how the fleet changes (rolling: N% at a time; blue-green: full fleet in new stack; canary: 2% first).
  • Each new pod passes readiness (dependencies wired), traffic adds after health check.
  • Progressive gates (canary analysis) keep or abort based on metric health in a window.
  • Abort → auto-rollback to the last-good image; green drained/removed.

Canary with analysis gate.

Deploy 5%
Analyze
Graduate
Gate fails
Steady
Hands-on

Examples

Argo Rollouts canary stub

Argo Rollouts canary stub
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: api }
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 60s }
        - analysis:
            templates: [{ spec: { metrics: [
              { name: error-rate, interval: 30s,
                successCondition: "result[0] < 0.5" } ] } }]
        - setWeight: 50
        - pause: { duration: 60s }
        - setWeight: 100

Traffic shifts 5→50→100 with an SLO gate between steps; a p95/error rate drop aborts instantly.

Real world

In production

  • All services: immutable images + rolling with readiness probes (K8s default).
  • High-risk services: canary 5% + auto-rollback on error-rate burn.
  • Big revision weeks: blue/green for zero-downtime DB-required swaps with migration steps.
Guidelines

Best practices

  • Readiness probe checks real dependencies — then rollouts only promote healthy pods.
  • Make migrations backward compatible (old image + new schema both work) for smooth rolling shifts.
  • Define an abort criteria BEFORE the rollout — post-hoc judging defeats the point.
  • Golden image + immutable tags: never mutate prod instances; rollback = repoint.
  • Run a release checklist: db-migrated, dashboards watched, revert ready, owner named.
Watch out

Common mistakes

  • Rolling into an unreadiness loop because the probe checks only / (not readiness to serve).
  • Canary percentage so low (or client-cache heavy) that analysis measures nothing.
  • Blue/green with long-lived sessions → session-loss blast on flip; solve for stateless or sticky-window.
  • No DB-migration compatibility → old pods 500 during the rollout window.
When things break

Debugging tips

  • Rollback doesn't fix 5xx — check if the DB migration is forward-incompatible (need migration revert too).
  • Canary looks healthy but p95 bad — your probe/analysis measures the request path as a whole, not the new revision's traffic selectors.
  • Rolling stuck at 0% — image pull credentials/workload identity broke for new nodes.
  • Green still serving post-flip — double-check LB weights / endpoint sets fully moved, and drain controller.
Interviews

Interview checkpoint

Test yourself

Quick quiz

0/2
1

Which strategy lets you revert with one command while keeping full capacity warm?

2

The #1 enabler of safe rolling deployments is:

Go deeper

Official documentation

K8s deployment strategies
docs
Argo Rollouts
docs
AWS CodeDeploy blue/green
docs