Lesson 4 of 7 in DevOps and delivery fundamentals
Environments: dev, staging and production
An environment is a separate place where the application runs, with its own servers, data and settings. Teams use several, commonly development, staging and production, so a change can be tried safely before real users depend on it.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: CI, continuous delivery and continuous deployment
The problem this solves
If the only place your app runs is the one real users use, every experiment is public. A bad change can break the service, and testing against real data can damage it.
Separate environments give you somewhere to be wrong cheaply. Development is for building. Staging is a rehearsal that should look as much like production as is practical. Production is where real users and real data live.
The risk is drift: environments slowly becoming different in ways nobody wrote down. The fix is to keep the build identical everywhere and move everything that must differ into configuration.
How it works
- Runs code
- People and clients
- Security
The same build is promoted from development to staging to production, while each environment reads its own settings from a separate configuration source.
- Development to Staging: 1 same build
- Staging to Production: 2 promote
- Production to Users: 3 serves
- Config to Development: settings
- Config to Staging: settings
- Config to Production: settings
Read the top row as the journey of one build. It is created once, tried in development, promoted to staging, and then promoted to production. Nothing is rebuilt along the way, so what you tested is what you ship.
The dashed arrows are configuration: database addresses, feature switches, credentials, log levels. They come from outside the build and differ per environment. Secrets such as passwords belong in a secrets manager or the platform's protected variables, never in the repository.
Staging earns its keep by matching production in the ways that cause failures, such as the same operating system, the same versions and a similar network layout. It is usually smaller and holds fake or anonymised data.
Try it yourself
You will run one script in three environments by changing only its settings. Linux, macOS or WSL. The host names are made up and nothing connects to them.
- 1Write one app that reads its settings from the environment
mkdir -p ~/devops-lab/environments && cd ~/devops-lab/environments cat > app.sh <<'EOF' [ -n "$DB_HOST" ] || { echo "DB_HOST is required" >&2; exit 1; } echo "env=$APP_ENV db=$DB_HOST debug=$DEBUG" EOFThe script is the build. It contains no host names. The first line makes it refuse to start when the setting is missing.
- 2Run it as development
APP_ENV=dev DB_HOST=localhost DEBUG=1 sh app.sh
You should see
env=dev db=localhost debug=1
- 3Run the same file as staging
APP_ENV=staging DB_HOST=db.staging.example DEBUG=0 sh app.sh
You should see
env=staging db=db.staging.example debug=0
Same file, different behaviour, because only the settings changed.
- 4Forget a setting and see it fail early
APP_ENV=production sh app.sh echo "exit=$?"
You should see
DB_HOST is required exit=1
Failing at startup with a clear message beats connecting to the wrong place. If this prints a db line instead, your shell already has
DB_HOSTset. - 5Keep each environment's settings in its own file
printf 'APP_ENV=staging\nDB_HOST=db.staging.example\nDEBUG=0\n' > staging.env (set -a; . ./staging.env; sh app.sh)
You should see
env=staging db=db.staging.example debug=0
The parentheses run this in a subshell, so the variables do not stay in your terminal. Never commit a file holding real production passwords.
What breaks, and where to look
It works in staging and fails in production with
connection refusedorpermission denied.- WHY IT HAPPENS
- Production differs in something staging does not model: network rules, credentials, data volume, or a service that exists in only one of them.
- CHECK FIRST
- Read the first error in the production logs. Then compare the two environments' settings by listing the names of the variables each one defines, and diff the lists.
A change passed in staging and behaves differently in production although the code is the same.
- WHY IT HAPPENS
- The artifact was rebuilt for each environment, and the rebuild pulled different dependency versions, so production runs something staging never saw.
- CHECK FIRST
- Compare the identifier of what is running in each place, for example the commit hash or image digest. If they differ, build once and promote that artifact.
Unexpected rows or deletions appear in the production database after someone tested locally.
- WHY IT HAPPENS
- A local configuration file contained the production connection string, so a development run wrote to real data.
- CHECK FIRST
- Print the database host the process used, for example
echo "$DB_HOST". Give each environment separate credentials, and give development credentials no access to production.
When not to use this
- A very small project can begin with production plus a preview deployment per change. Every extra environment costs money and effort to keep in sync, so add them when a real risk justifies it.
- Staging never matches production exactly. Treat a pass there as evidence, not proof, and pair it with monitoring and gradual rollouts in production.
Check yourself
Why build once and promote the same artifact?
Rebuilding per environment can produce different results, so what was tested may not be what runs in production.
What belongs in configuration rather than in the build?
Anything that differs by environment, such as database addresses, credentials, feature switches and log levels.
What is environment drift?
Environments slowly becoming different in undocumented ways, so a test in one says little about another.
Keep going
Lesson 4 of 92 in the DevOps path