Lesson 2 of 7 in Python and scripting for ops
Python environments: venv and requirements
A virtual environment is a folder that holds its own copy of the Python interpreter links and its own installed packages. A requirements file lists the packages a project needs so someone else can rebuild the same environment.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Bash or Python: picking the right tool
The problem this solves
Your operating system uses Python for its own tools. If you install packages into that system Python, you can break those tools, and you can break one project by upgrading a package for another.
Two scripts on the same machine often need different versions of the same library. With one shared install, only one of them can be right.
A virtual environment gives each project its own package folder. A requirements file records what is in it, so a teammate, a CI job or a server can build the same thing from scratch.
How it works
- Other
- Data
The system Python creates a separate environment for each project, and each environment holds only that project's packages.
- System Python to Project A .venv: 1 python -m venv
- System Python to Project B .venv: 1 python -m venv
- Control
- Data
- Runs code
pip reads the list in requirements.txt, downloads each package from PyPI, installs them into the environment, and your script imports them from there.
- requirements.txt to pip: 1 read list
- pip to PyPI: 2 download
- pip to .venv: 3 install
- Your script to .venv: 4 import from
python3 -m venv .venv creates a folder named .venv in your project. Inside it is a bin folder with its own python and pip, and a site-packages folder where installed packages go.
Activating the environment with source .venv/bin/activate puts that bin folder first on your PATH, so the commands python and pip now mean the environment's copies. Activation is a convenience. You can also run .venv/bin/python directly, which is what cron jobs and CI should do.
pip freeze prints every installed package with its exact version. Saving that into requirements.txt lets you recreate the environment with pip install -r requirements.txt.
A virtual environment is not portable. Scripts inside it contain the absolute path of the interpreter, so moving or copying the folder breaks it. Delete it and rebuild from the requirements file instead.
Try it yourself
Needs python3 with the venv module and internet access to download from PyPI, which is free. Work inside the lab folder from the last concept.
- 1Create the environment
cd ~/scripting-lab python3 -m venv .venv
This makes a
.venvfolder. Nothing outside it changes. - 2Activate it and confirm you are using it
source .venv/bin/activate which python
The path printed should end in
scripting-lab/.venv/bin/python. Your prompt usually gains a(.venv)prefix too. - 3Install two libraries
python -m pip install requests PyYAML
Using
python -m pipguarantees pip belongs to the Python you just checked. Later concepts use these two libraries. - 4Record the environment
pip freeze > requirements.txt cat requirements.txt
Each line looks like
name==version. It includes packages thatrequestspulled in for itself. - 5Delete the environment and rebuild it from the file
deactivate cd ~/scripting-lab rm -rf .venv python3 -m venv .venv .venv/bin/python -m pip install -r requirements.txt .venv/bin/python -c "import requests, yaml; print('imports ok')"You should see
imports ok
The
rm -rfonly touches the.venvfolder in the lab. This is the whole point: the environment is disposable because the file can recreate it.
What breaks, and where to look
error: externally-managed-environment
- WHY IT HAPPENS
- You ran
pip installagainst the system Python on a recent Debian, Ubuntu or Homebrew setup. Those systems block it on purpose so you do not break OS tools. - CHECK FIRST
- Create and activate a virtual environment first, then run
python -m pip install ...inside it. Do not use--break-system-packagesfor project work.
ModuleNotFoundError: No module named 'requests'
- WHY IT HAPPENS
- The package is installed in the environment, but the script ran with a different Python. This is common in cron and CI where nothing activated the environment.
- CHECK FIRST
- Print the interpreter that is running:
python -c "import sys; print(sys.executable)". In cron, call/full/path/to/.venv/bin/python script.py.
The virtual environment was not created successfully because ensurepip is not available
- WHY IT HAPPENS
- On Debian and Ubuntu the
venvmodule ships in a separate package. - CHECK FIRST
- Install it with
sudo apt install python3-venv, then runpython3 -m venv .venvagain.
When not to use this
- A virtual environment isolates packages, not the Python version itself. If you need a specific Python version that your system does not have, use a version manager such as
pyenvor a tool such asuv. - For anything you deploy, pin versions in the requirements file and build inside a container or fresh CI job. A file made by
pip freezeon your laptop records your laptop's choices, which may not suit another operating system.
Check yourself
Why not just pip install into the system Python?
It can break tools the operating system depends on, and two projects that need different versions of one library cannot both be satisfied.
Why can a cron job fail with ModuleNotFoundError even though you installed the package?
Cron did not activate your environment, so it ran the system Python. Call the environment's own python by full path.
What do you do instead of moving a virtual environment to a new folder?
Delete it and rebuild it from requirements.txt, because the scripts inside hold absolute paths to the old location.
Keep going
Lesson 25 of 92 in the DevOps path