Lesson 2 of 7 in Python and scripting for ops

Python environments: venv and requirements

Core8 min

A virtual environment is a folder that holds its own copy of the Python interpreter links and its own installed packages. A requirements file lists the packages a project needs so someone else can rebuild the same environment.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Bash or Python: picking the right tool

The problem this solves

Your operating system uses Python for its own tools. If you install packages into that system Python, you can break those tools, and you can break one project by upgrading a package for another.

Two scripts on the same machine often need different versions of the same library. With one shared install, only one of them can be right.

A virtual environment gives each project its own package folder. A requirements file records what is in it, so a teammate, a CI job or a server can build the same thing from scratch.

How it works

One system Python, one private environment per project
One system Python, one private environment per projectThe system Python creates a separate environment for each project, and each environment holds only that project's packages.1 python -m venv1 python -m venvSystem Pythonowned by the OSProject A .venvits own packagesProject B .venvits own packages
  • Other
  • Data

The system Python creates a separate environment for each project, and each environment holds only that project's packages.

  1. System Python to Project A .venv: 1 python -m venv
  2. System Python to Project B .venv: 1 python -m venv
How requirements.txt rebuilds an environment
How requirements.txt rebuilds an environmentpip reads the list in requirements.txt, downloads each package from PyPI, installs them into the environment, and your script imports them from there.1 read list2 download3 install4 import fromrequirements.txtthe package listpipthe installerPyPIpublic package index.venvinstalled packagesYour scriptimport requests
  • Control
  • Data
  • Runs code

pip reads the list in requirements.txt, downloads each package from PyPI, installs them into the environment, and your script imports them from there.

  1. requirements.txt to pip: 1 read list
  2. pip to PyPI: 2 download
  3. pip to .venv: 3 install
  4. Your script to .venv: 4 import from

python3 -m venv .venv creates a folder named .venv in your project. Inside it is a bin folder with its own python and pip, and a site-packages folder where installed packages go.

Activating the environment with source .venv/bin/activate puts that bin folder first on your PATH, so the commands python and pip now mean the environment's copies. Activation is a convenience. You can also run .venv/bin/python directly, which is what cron jobs and CI should do.

pip freeze prints every installed package with its exact version. Saving that into requirements.txt lets you recreate the environment with pip install -r requirements.txt.

A virtual environment is not portable. Scripts inside it contain the absolute path of the interpreter, so moving or copying the folder breaks it. Delete it and rebuild from the requirements file instead.

Try it yourself

Needs python3 with the venv module and internet access to download from PyPI, which is free. Work inside the lab folder from the last concept.

  1. 1
    Create the environment
    cd ~/scripting-lab
    python3 -m venv .venv

    This makes a .venv folder. Nothing outside it changes.

  2. 2
    Activate it and confirm you are using it
    source .venv/bin/activate
    which python

    The path printed should end in scripting-lab/.venv/bin/python. Your prompt usually gains a (.venv) prefix too.

  3. 3
    Install two libraries
    python -m pip install requests PyYAML

    Using python -m pip guarantees pip belongs to the Python you just checked. Later concepts use these two libraries.

  4. 4
    Record the environment
    pip freeze > requirements.txt
    cat requirements.txt

    Each line looks like name==version. It includes packages that requests pulled in for itself.

  5. 5
    Delete the environment and rebuild it from the file
    deactivate
    cd ~/scripting-lab
    rm -rf .venv
    python3 -m venv .venv
    .venv/bin/python -m pip install -r requirements.txt
    .venv/bin/python -c "import requests, yaml; print('imports ok')"

    You should see

    imports ok

    The rm -rf only touches the .venv folder in the lab. This is the whole point: the environment is disposable because the file can recreate it.

What breaks, and where to look

  • error: externally-managed-environment

    WHY IT HAPPENS
    You ran pip install against the system Python on a recent Debian, Ubuntu or Homebrew setup. Those systems block it on purpose so you do not break OS tools.
    CHECK FIRST
    Create and activate a virtual environment first, then run python -m pip install ... inside it. Do not use --break-system-packages for project work.
  • ModuleNotFoundError: No module named 'requests'

    WHY IT HAPPENS
    The package is installed in the environment, but the script ran with a different Python. This is common in cron and CI where nothing activated the environment.
    CHECK FIRST
    Print the interpreter that is running: python -c "import sys; print(sys.executable)". In cron, call /full/path/to/.venv/bin/python script.py.
  • The virtual environment was not created successfully because ensurepip is not available

    WHY IT HAPPENS
    On Debian and Ubuntu the venv module ships in a separate package.
    CHECK FIRST
    Install it with sudo apt install python3-venv, then run python3 -m venv .venv again.

When not to use this

  • A virtual environment isolates packages, not the Python version itself. If you need a specific Python version that your system does not have, use a version manager such as pyenv or a tool such as uv.
  • For anything you deploy, pin versions in the requirements file and build inside a container or fresh CI job. A file made by pip freeze on your laptop records your laptop's choices, which may not suit another operating system.

Check yourself

Why not just pip install into the system Python?

It can break tools the operating system depends on, and two projects that need different versions of one library cannot both be satisfied.

Why can a cron job fail with ModuleNotFoundError even though you installed the package?

Cron did not activate your environment, so it ran the system Python. Call the environment's own python by full path.

What do you do instead of moving a virtual environment to a new folder?

Delete it and rebuild it from requirements.txt, because the scripts inside hold absolute paths to the old location.

Lesson 25 of 92 in the DevOps path

Search

Search cheat sheets and projects