Lesson 4 of 7 in DevOps and delivery fundamentals

Environments: dev, staging and production

Core8 min

An environment is a separate place where the application runs, with its own servers, data and settings. Teams use several, commonly development, staging and production, so a change can be tried safely before real users depend on it.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: CI, continuous delivery and continuous deployment

The problem this solves

If the only place your app runs is the one real users use, every experiment is public. A bad change can break the service, and testing against real data can damage it.

Separate environments give you somewhere to be wrong cheaply. Development is for building. Staging is a rehearsal that should look as much like production as is practical. Production is where real users and real data live.

The risk is drift: environments slowly becoming different in ways nobody wrote down. The fix is to keep the build identical everywhere and move everything that must differ into configuration.

How it works

One build moves up through environments, config differs
One build moves up through environments, config differsThe same build is promoted from development to staging to production, while each environment reads its own settings from a separate configuration source.1 same build2 promote3 servessettingssettingssettingsDevelopmentbuild and experimentStagingrehearsalProductionreal dataUsersConfigper-environment values
  • Runs code
  • People and clients
  • Security

The same build is promoted from development to staging to production, while each environment reads its own settings from a separate configuration source.

  1. Development to Staging: 1 same build
  2. Staging to Production: 2 promote
  3. Production to Users: 3 serves
  4. Config to Development: settings
  5. Config to Staging: settings
  6. Config to Production: settings

Read the top row as the journey of one build. It is created once, tried in development, promoted to staging, and then promoted to production. Nothing is rebuilt along the way, so what you tested is what you ship.

The dashed arrows are configuration: database addresses, feature switches, credentials, log levels. They come from outside the build and differ per environment. Secrets such as passwords belong in a secrets manager or the platform's protected variables, never in the repository.

Staging earns its keep by matching production in the ways that cause failures, such as the same operating system, the same versions and a similar network layout. It is usually smaller and holds fake or anonymised data.

Try it yourself

You will run one script in three environments by changing only its settings. Linux, macOS or WSL. The host names are made up and nothing connects to them.

  1. 1
    Write one app that reads its settings from the environment
    mkdir -p ~/devops-lab/environments && cd ~/devops-lab/environments
    cat > app.sh <<'EOF'
    [ -n "$DB_HOST" ] || { echo "DB_HOST is required" >&2; exit 1; }
    echo "env=$APP_ENV db=$DB_HOST debug=$DEBUG"
    EOF

    The script is the build. It contains no host names. The first line makes it refuse to start when the setting is missing.

  2. 2
    Run it as development
    APP_ENV=dev DB_HOST=localhost DEBUG=1 sh app.sh

    You should see

    env=dev db=localhost debug=1
  3. 3
    Run the same file as staging
    APP_ENV=staging DB_HOST=db.staging.example DEBUG=0 sh app.sh

    You should see

    env=staging db=db.staging.example debug=0

    Same file, different behaviour, because only the settings changed.

  4. 4
    Forget a setting and see it fail early
    APP_ENV=production sh app.sh
    echo "exit=$?"

    You should see

    DB_HOST is required
    exit=1

    Failing at startup with a clear message beats connecting to the wrong place. If this prints a db line instead, your shell already has DB_HOST set.

  5. 5
    Keep each environment's settings in its own file
    printf 'APP_ENV=staging\nDB_HOST=db.staging.example\nDEBUG=0\n' > staging.env
    (set -a; . ./staging.env; sh app.sh)

    You should see

    env=staging db=db.staging.example debug=0

    The parentheses run this in a subshell, so the variables do not stay in your terminal. Never commit a file holding real production passwords.

What breaks, and where to look

  • It works in staging and fails in production with connection refused or permission denied.

    WHY IT HAPPENS
    Production differs in something staging does not model: network rules, credentials, data volume, or a service that exists in only one of them.
    CHECK FIRST
    Read the first error in the production logs. Then compare the two environments' settings by listing the names of the variables each one defines, and diff the lists.
  • A change passed in staging and behaves differently in production although the code is the same.

    WHY IT HAPPENS
    The artifact was rebuilt for each environment, and the rebuild pulled different dependency versions, so production runs something staging never saw.
    CHECK FIRST
    Compare the identifier of what is running in each place, for example the commit hash or image digest. If they differ, build once and promote that artifact.
  • Unexpected rows or deletions appear in the production database after someone tested locally.

    WHY IT HAPPENS
    A local configuration file contained the production connection string, so a development run wrote to real data.
    CHECK FIRST
    Print the database host the process used, for example echo "$DB_HOST". Give each environment separate credentials, and give development credentials no access to production.

When not to use this

  • A very small project can begin with production plus a preview deployment per change. Every extra environment costs money and effort to keep in sync, so add them when a real risk justifies it.
  • Staging never matches production exactly. Treat a pass there as evidence, not proof, and pair it with monitoring and gradual rollouts in production.

Check yourself

Why build once and promote the same artifact?

Rebuilding per environment can produce different results, so what was tested may not be what runs in production.

What belongs in configuration rather than in the build?

Anything that differs by environment, such as database addresses, credentials, feature switches and log levels.

What is environment drift?

Environments slowly becoming different in undocumented ways, so a test in one says little about another.

Lesson 4 of 92 in the DevOps path

Search

Search cheat sheets and projects