Lesson 4 of 7 in Infrastructure as Code with Terraform and OpenTofu

State, drift and locking

Core13 min

State is a file the tool keeps to remember which real objects belong to which resources in your configuration. Locking stops two runs from writing it at the same time. Drift is when the real world no longer matches what the configuration and state expect.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: The workflow: init, plan, apply, destroy

The problem this solves

Your files say what you want. They do not say which real server is the one your resource block created. The tool needs a memory that links each block to a real object, otherwise it could not update or delete anything safely.

That memory is the state file. It is also a problem, because it can contain secrets in plain text, and because two people running apply at once could corrupt it or create duplicates.

So teams keep state out of Git, store it in a shared remote backend with access control, and rely on locking so only one run writes at a time.

How it works

Three things the tool compares
Three things the tool comparesThe tool reads your config, reads its state, and reads the real world, then decides what to change.1 desired2 read and update3 refresh andchangeConfigwhat you wantterraform / tofuplans the diffReal worldwhat exists nowStatewhat it remembers
  • Control
  • Runs code
  • Other
  • Data

The tool reads your config, reads its state, and reads the real world, then decides what to change.

  1. Config to terraform / tofu: 1 desired
  2. terraform / tofu to State: 2 read and update
  3. terraform / tofu to Real world: 3 refresh and change
Two people, one remote state, one lock
Two people, one remote state, one lockPerson A takes the lock and writes state. Person B tries to run at the same time and is refused until the lock is released.1 take lock, write state2 try lock: refusedPerson Aruns applyPerson Bruns plan at same timeRemote backendstate plus a lock
  • People and clients
  • Data

Person A takes the lock and writes state. Person B tries to run at the same time and is refused until the lock is released.

  1. Person A to Remote backend: 1 take lock, write state
  2. Person B to Remote backend: 2 try lock: refused

Before it plans, the tool refreshes: it asks the provider what really exists for every object in state. It then compares your config, that fresh view and the state. The difference is the plan.

By default state is a local file called terraform.tfstate. That is fine for learning and unsafe for teams. A remote backend stores state somewhere shared, such as an object storage bucket, and a good backend also supports locking. Not every backend does, so check the backend's documentation.

With locking, any command that could write state first takes a lock. A second run gets an error that names who holds it, instead of silently racing. In current Terraform the S3 backend can keep a lock file next to the state when you set use_lockfile. Older setups used a DynamoDB table, which the Terraform documentation now marks as deprecated.

Drift happens when someone changes a real object outside the tool. The next plan notices because of the refresh. You then choose: apply to put the object back, or edit the config to accept the new reality. terraform plan -refresh-only shows drift without proposing changes to your config.

Try it yourself

You will inspect a real state file, see a secret in it, create drift on purpose and trigger a lock error. All with local providers, in a throwaway folder.

  1. 1
    Create something to track
    mkdir -p ~/iac-lab/state && cd ~/iac-lab/state
    cat > main.tf <<"EOF"
    terraform {
      required_providers {
        local  = { source = "hashicorp/local" }
        random = { source = "hashicorp/random" }
      }
    }
    
    resource "random_password" "db" {
      length = 16
    }
    
    resource "local_file" "greeting" {
      filename = "greeting.txt"
      content  = "state demo"
    }
    EOF
    terraform init
    terraform apply -auto-approve
  2. 2
    Ask the tool what it remembers
    terraform state list
    terraform state show local_file.greeting

    The list shows resource addresses such as local_file.greeting. Each address is the link between a block in your file and a real object.

  3. 3
    Look inside the file and protect it
    grep -n '"result"' terraform.tfstate
    printf '.terraform/\n*.tfstate\n*.tfstate.backup\n' > .gitignore

    The generated password appears in plain text in the state file. That is why .gitignore excludes state. Do commit .terraform.lock.hcl, but never state.

  4. 4
    Create drift by hand
    rm greeting.txt
    terraform plan

    You changed the real world behind the tool's back. The plan notices and proposes to create local_file.greeting again. Do not apply yet.

  5. 5
    Trigger a lock error
    # Terminal A: leave this waiting at the yes/no prompt
    terraform apply
    
    # Terminal B, same folder, while A waits:
    terraform plan

    Terminal B fails with an error starting Error acquiring the state lock, because A holds the lock while it waits for your answer. Type no in A to release it. Then clean up with terraform destroy -auto-approve.

What breaks, and where to look

  • Error: Error acquiring the state lock, with Lock Info showing an ID, an operation and a who/when.

    WHY IT HAPPENS
    Another run really holds the lock, or an earlier run was killed (closed laptop, cancelled pipeline) and left a stale lock on a remote backend.
    CHECK FIRST
    First find out whether someone is still running: ask the team or check pipelines. If the run is truly gone, terraform force-unlock <LOCK_ID> with the ID from the error. Use it only for a lock you know is dead, because unlocking a live run allows two writers.
  • The plan wants to create everything again, even resources that already exist, and apply then fails with already exists errors.

    WHY IT HAPPENS
    The tool is looking at an empty or wrong state: a new machine without the state file, a different backend or workspace, or a deleted terraform.tfstate.
    CHECK FIRST
    Run terraform state list. If it is empty, check the backend block, terraform workspace show, and whether state was restored. To adopt existing objects use terraform import or an import block.
  • The plan shows changes in a resource nobody edited in the code.

    WHY IT HAPPENS
    Drift: someone edited the object in a console or another tool, or a value is changed by the platform itself.
    CHECK FIRST
    Run terraform plan -refresh-only to see only the drift. Then either apply to restore it or update the config to match.

When not to use this

  • Never use -lock=false just to get past a lock error. It removes the protection that prevents corrupted state.
  • Hand-editing the state file is a last resort. Use terraform state subcommands, and take a copy first.
  • Locking protects state writes. It does not stop two different configurations from fighting over the same real object.

Check yourself

Why should the state file not be committed to Git?

It can contain secrets in plain text, and Git has no locking, so two people can overwrite each other.

What is drift and how do you see it?

Drift is a real object changing outside the tool. A plan, or plan -refresh-only, shows it after refreshing.

When is force-unlock acceptable?

Only when you are sure the run holding the lock is dead, using the lock ID from the error.

Lesson 66 of 92 in the DevOps path

Search

Search cheat sheets and projects