Terraform
Advanced 15 min read Terraform

Terraform in Practice: Workflows

The daily rhythm of infra-as-code in production

Production Terraform is a relationship: a mono-repo of stack dirs, per-env backends, plan-gated applies, drift scans, terraform-docs/READMEs, and releases. Learn the daily rhythm — plan, review, apply, verify, and the ops side (drift, state surgery, catastrophic recovery).

workflowmonorepodriftstate surgeryapply orderingrecoveryreleases
Prerequisites: All prior Terraform concepts
Before we start

What you will learn

  • Design a repo layout: stacks, modules, per-env backends, releases
  • Run plan/apply with review gates and drift scans
  • Execute state surgery and recovery when things go wrong
In 60 seconds

Repos mix modules (reusable) and stacks (environments). Each stack owns its backend; CI plans + reviews; apply follows. Drift scans run on cron; recovery drills use versioned state + import. Terraform is 20% language, 80% workflow discipline.

Introduction

What is this?

Repo shape: infra/ (monorepo) with modules/, envs/, stacks/ inside; or per-team repos referencing a module registry. Stacks: smallest deployable unit with its own backend. CI: per-stack jobs, plan-comment + apply gate, dependency ordering (network → data → app). Drift: scheduled plan jobs post diffs when reality drifted; tool runners (atlantis, tf-docs, pre-commit terraform_fmt, checkov). Releases: version bumps for modules, release branches if pinned, promote via vars not widgets. State surgery: import, mv, rm; recovery: restore a version, or pragmatic re-create with plan-proof; catastrophic: out-of-band manual recovery + reconcile with import. Operate-in-bands: never mix TFC run queue with external bots without sealing them.

Motivation

Why do you need it?

  • A clean repo layout makes 50 stacks reviewable instead of a monolith myth.
  • Apply gates + drift scans keep 'who changed prod' answerable Monday-morning.
  • State surgery skills are the difference between a fix and a 3am recreate.
  • Versioned releases mean rollback is a pin, not new archaeology.
Think about it

Mental model

Imagine

Running a fleet of stores with a central recipe book, regional managers, and weekly health checklists

Think of

Terraform practice

One recipe book (modules) across stores (stacks); regional managers (CI/approvals) run the daily open/close (apply/destroy); the weekly checklist (drift scan) flags shelves that moved (drift).

Mechanics

How it works

  • Merge → CI builds stacks, plans them, posts to the PR; reviewers approve.
  • Apply job re-runs plan, checks it matches, applies within backend lock.
  • Cron drift job plans all envs and alerts on diffs (reality ≠ code).
  • State surgery uses import/mv/rm for moves; rollback reads a prior state version.
  • Catastrophic recovery: rebuild via code + re-import managed bits, verified by plan.

Production Terraform day.

PR
Plan
Apply gate
Verify
Drift scan
Recovery
Hands-on

Examples

Repo layout that scales

Repo layout that scales
infra/
  modules/           # reusable, versioned
    vpc/ tf-aws-vpc/ labels/
  stacks/
    network/prod/ backend.tf main.tf prod.auto.tfvars
    network/dev/...
    data/prod/...
    app/prod/...
  scripts/plan.sh apply.sh drift-report.sh
  .github/workflows/plan.yml apply.yml drift.yml

Modules are versioned units; stacks are deployable envs; CI orchestrates plan→review→apply; drift reports keep reality honest.

Real world

In production

  • Weekly drift report pings SRE channel with per-stack plan diffs.
  • Module release bumps rolled out via PR + plan comment across consumers.
  • 3am flapping state surgery: import + mv + a reviewed plan, never an ad-hoc state rm.
  • Rollback of a bad apply: restore S3 state-version then plan (or reuse the prior tfplan).
Guidelines

Best practices

  • Give each stack an owner + Slack diffusion for drift alerts.
  • Separate apply responsibility: humans review prod, pipeline handles dev.
  • Keep modules lean; document outputs as the dependency surface.
  • Drill recovery quarterly: kill a state version, restore, import, and confirm plan-zeros.
  • Never mix human run + bot run on the same state key simultaneously.
Watch out

Common mistakes

  • One giant state file for everything → every PR's plan is global and reviewer-blind.
  • State surgery with zero plan review (rm is forever if un-imported).
  • Drift alerts that nobody acts on → Ctrl-c silence until prod breaks.
  • Releasing modules without version pins → downstream stacks auto-jump versions.
When things break

Debugging tips

  • Plan differs from the approved one → pin/apply timing; re-plan at apply job and diff — don't apply blind.
  • Drift appearing constant → likely in-band changes logging; identify the change source via recent apply audit.
  • State version restore left a wrong provider view → run refresh then inspect plan before fixing.
  • Import loops → check for duplicates in state; mv before importing again.
Interviews

Interview checkpoint

Practice

Hands-on lab

Shred-Proof Recovery: Restore + Import Drill
45 min · advanced
In progress
Objectives
  • Simulate a corrupted state and recover from a versioned snapshot
  • Rebuild state for an orphaned resource via import
  • Converge a recovered stack to plan-zero

Shred-Proof Recovery: Restore + Import Drill

45 min · advanced

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

Best reply when prod state version is corrupt:

2

Drift scan in CI's purpose:

Go deeper

Official documentation

Terraform CLI workflows
docs
Run tasks / policy in Cloud
docs
Community: atlantis
docs