Lesson 5 of 7 in Python and scripting for ops
Running commands with subprocess
The subprocess module starts another program from your script and hands back its output and exit code. Passing the command as a list of separate arguments, without a shell, keeps untrusted text from being interpreted as commands.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Bash or Python: picking the right tool
The problem this solves
Ops scripts often need a tool that already exists: git, kubectl, df, systemctl. Python can start those programs, but you have to decide how to hand them arguments, how to read what they print, and what to do when they fail.
The easy way, building one long string and running it through a shell, has a sharp edge. If any part of that string comes from outside your script, such as a hostname typed by a user or read from a file, a value like x; rm -rf something can become a second command.
The other common failure is silence. Without a check, a failed command returns a number and the script carries on as if it worked.
How it works
- Runs code
- Control
- Other
Your script calls subprocess.run with a list of arguments, which starts the program directly, and the result with its exit code and output comes back for the script to check.
- Your script to subprocess.run: 1 call
- subprocess.run to Program: 2 start
- Program to Result: 3 exit and output
- Result to Your script: 4 check returncode
- People and clients
- Runs code
- Security
- Control
With shell=True the shell splits untrusted text and can run an extra command, while with a list the text is passed to the program as one plain argument.
- Untrusted text to Shell parses it: 1 shell=True
- Shell parses it to Extra command: 2 splits and runs
- Untrusted text to List of args: 1 list form
- List of args to Program: 2 passed as data
subprocess.run(["df", "-h", "/"], capture_output=True, text=True, check=True, timeout=10) starts df directly. Each list item is exactly one argument, so spaces and punctuation inside an item are just characters.
capture_output=True collects what the program prints. text=True gives you strings instead of bytes. check=True raises CalledProcessError on a non-zero exit code, so a failure cannot slip by. timeout stops waiting after that many seconds and raises TimeoutExpired.
shell=True runs your string through a shell. That is useful when you really want pipes and wildcards, but the shell interprets characters like ;, | and $(. If any of the text is not fully under your control, it becomes an injection hole.
If the program is not installed, subprocess.run raises FileNotFoundError before anything runs. That is different from a program that starts and then fails, which gives an exit code.
Try it yourself
Runs on Linux or macOS. The commands only read information or print text. Run them from any folder.
- 1Run a command and read its result
python3 - <<'EOF' import subprocess r = subprocess.run(["df", "-h", "/"], capture_output=True, text=True, timeout=10) print(r.returncode) print(r.stdout.splitlines()[0]) EOF
The first line printed is 0. The second is the header row of
df, which looks a little different on Linux and macOS. - 2Turn a failure into an exception
python3 - <<'EOF' import subprocess try: subprocess.run(["ls", "/does-not-exist"], capture_output=True, text=True, check=True) except subprocess.CalledProcessError as e: print("exit code:", e.returncode) print("stderr:", e.stderr.strip()) EOFThe exit code is 2 with GNU
lson Linux and 1 on macOS.e.stderrholds the message the program printed, which is the first thing to log. - 3Compare shell=True with a list
python3 - <<'EOF' import subprocess text = "hello; echo INJECTED" a = subprocess.run(f"echo {text}", shell=True, capture_output=True, text=True) b = subprocess.run(["echo", text], capture_output=True, text=True) print("shell=True:", a.stdout.splitlines()) print("list args :", b.stdout.splitlines()) EOFYou should see
shell=True: ['hello', 'INJECTED'] list args : ['hello; echo INJECTED']
With the shell, the text after the semicolon ran as a second command. The harmless
echostands in for anything an attacker would choose. - 4Stop waiting after a timeout
python3 - <<'EOF' import subprocess try: subprocess.run(["sleep", "5"], timeout=1) except subprocess.TimeoutExpired as e: print("gave up after", e.timeout, "second") EOFPython kills the child process when the timeout expires, then raises the exception.
What breaks, and where to look
FileNotFoundError: [Errno 2] No such file or directory: 'ls -l'
- WHY IT HAPPENS
- You passed a whole command line as one string without
shell=True. Python looked for a program literally namedls -l. - CHECK FIRST
- Split it into a list,
["ls", "-l"]. For a long string you did not write,shlex.split(...)splits it the way a shell would, without running anything.
The script hangs and prints nothing, and Ctrl+C is the only way out.
- WHY IT HAPPENS
- The child program is waiting, often for input such as a password or a yes/no prompt, or it is just slow, and no timeout was set.
- CHECK FIRST
- Add
timeout=andstdin=subprocess.DEVNULLto the call. In a second terminal,ps aux | grep <program>shows whether the child is still running.
The script finishes with no error, but the file or deployment it should have produced is missing.
- WHY IT HAPPENS
- A command failed, but the code never looked at
returncode, so the script kept going. Another version of this bug isTypeError: a bytes-like object is required, not 'str', which meanstext=Trueis missing and the output is bytes. - CHECK FIRST
- Add
check=Trueand logstderrin theexcept subprocess.CalledProcessErrorhandler. Addtext=Trueto get strings.
When not to use this
- Do not shell out for something Python already does. Use
pathlibandshutilfor files,requestsfor HTTP, andshutil.disk_usagefor disk space. They return real values instead of text you must parse. - If the script is only a chain of a few tools joined by pipes, a short Bash script is clearer than building the same thing out of several
subprocesscalls. subprocess.runwaits for the program to finish. For a program that runs for hours or streams output as it goes, you needsubprocess.Popen, which is a more advanced tool.
Check yourself
Why is subprocess.run(["echo", user_text]) safer than subprocess.run(f"echo {user_text}", shell=True)?
With a list there is no shell, so characters like ; and | in the text stay plain data instead of starting another command.
What does check=True do?
It raises CalledProcessError when the program exits with a non-zero code, so the failure cannot be ignored by accident.
Which exception means the program was not found, and which means it ran and failed?
FileNotFoundError means it could not be started. CalledProcessError (with check=True) means it started and returned a non-zero exit code.
Keep going
Lesson 28 of 92 in the DevOps path