Lesson 5 of 7 in Python and scripting for ops

Running commands with subprocess

Core9 min

The subprocess module starts another program from your script and hands back its output and exit code. Passing the command as a list of separate arguments, without a shell, keeps untrusted text from being interpreted as commands.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Bash or Python: picking the right tool

The problem this solves

Ops scripts often need a tool that already exists: git, kubectl, df, systemctl. Python can start those programs, but you have to decide how to hand them arguments, how to read what they print, and what to do when they fail.

The easy way, building one long string and running it through a shell, has a sharp edge. If any part of that string comes from outside your script, such as a hostname typed by a user or read from a file, a value like x; rm -rf something can become a second command.

The other common failure is silence. Without a check, a failed command returns a number and the script carries on as if it worked.

How it works

What subprocess.run does for you
What subprocess.run does for youYour script calls subprocess.run with a list of arguments, which starts the program directly, and the result with its exit code and output comes back for the script to check.1 call2 start3 exit and output4 check returncodeYour scriptdecides what to runsubprocess.runlist of args, no shellProgramgit, df, systemctlResultreturncode, stdout, stderr
  • Runs code
  • Control
  • Other

Your script calls subprocess.run with a list of arguments, which starts the program directly, and the result with its exit code and output comes back for the script to check.

  1. Your script to subprocess.run: 1 call
  2. subprocess.run to Program: 2 start
  3. Program to Result: 3 exit and output
  4. Result to Your script: 4 check returncode
Why a list of arguments is safer than shell=True
Why a list of arguments is safer than shell=TrueWith shell=True the shell splits untrusted text and can run an extra command, while with a list the text is passed to the program as one plain argument.1 shell=True2 splits andruns1 list form2 passed asdataUntrusted texthello; echo INJECTEDShell parses itsplits on ; | $( )Extra commandruns with your rightsList of argsone item, no parsingProgramgets the text as data
  • People and clients
  • Runs code
  • Security
  • Control

With shell=True the shell splits untrusted text and can run an extra command, while with a list the text is passed to the program as one plain argument.

  1. Untrusted text to Shell parses it: 1 shell=True
  2. Shell parses it to Extra command: 2 splits and runs
  3. Untrusted text to List of args: 1 list form
  4. List of args to Program: 2 passed as data

subprocess.run(["df", "-h", "/"], capture_output=True, text=True, check=True, timeout=10) starts df directly. Each list item is exactly one argument, so spaces and punctuation inside an item are just characters.

capture_output=True collects what the program prints. text=True gives you strings instead of bytes. check=True raises CalledProcessError on a non-zero exit code, so a failure cannot slip by. timeout stops waiting after that many seconds and raises TimeoutExpired.

shell=True runs your string through a shell. That is useful when you really want pipes and wildcards, but the shell interprets characters like ;, | and $(. If any of the text is not fully under your control, it becomes an injection hole.

If the program is not installed, subprocess.run raises FileNotFoundError before anything runs. That is different from a program that starts and then fails, which gives an exit code.

Try it yourself

Runs on Linux or macOS. The commands only read information or print text. Run them from any folder.

  1. 1
    Run a command and read its result
    python3 - <<'EOF'
    import subprocess
    
    r = subprocess.run(["df", "-h", "/"], capture_output=True, text=True, timeout=10)
    print(r.returncode)
    print(r.stdout.splitlines()[0])
    EOF

    The first line printed is 0. The second is the header row of df, which looks a little different on Linux and macOS.

  2. 2
    Turn a failure into an exception
    python3 - <<'EOF'
    import subprocess
    
    try:
        subprocess.run(["ls", "/does-not-exist"], capture_output=True, text=True, check=True)
    except subprocess.CalledProcessError as e:
        print("exit code:", e.returncode)
        print("stderr:", e.stderr.strip())
    EOF

    The exit code is 2 with GNU ls on Linux and 1 on macOS. e.stderr holds the message the program printed, which is the first thing to log.

  3. 3
    Compare shell=True with a list
    python3 - <<'EOF'
    import subprocess
    
    text = "hello; echo INJECTED"
    a = subprocess.run(f"echo {text}", shell=True, capture_output=True, text=True)
    b = subprocess.run(["echo", text], capture_output=True, text=True)
    print("shell=True:", a.stdout.splitlines())
    print("list args :", b.stdout.splitlines())
    EOF

    You should see

    shell=True: ['hello', 'INJECTED']
    list args : ['hello; echo INJECTED']

    With the shell, the text after the semicolon ran as a second command. The harmless echo stands in for anything an attacker would choose.

  4. 4
    Stop waiting after a timeout
    python3 - <<'EOF'
    import subprocess
    
    try:
        subprocess.run(["sleep", "5"], timeout=1)
    except subprocess.TimeoutExpired as e:
        print("gave up after", e.timeout, "second")
    EOF

    Python kills the child process when the timeout expires, then raises the exception.

What breaks, and where to look

  • FileNotFoundError: [Errno 2] No such file or directory: 'ls -l'

    WHY IT HAPPENS
    You passed a whole command line as one string without shell=True. Python looked for a program literally named ls -l.
    CHECK FIRST
    Split it into a list, ["ls", "-l"]. For a long string you did not write, shlex.split(...) splits it the way a shell would, without running anything.
  • The script hangs and prints nothing, and Ctrl+C is the only way out.

    WHY IT HAPPENS
    The child program is waiting, often for input such as a password or a yes/no prompt, or it is just slow, and no timeout was set.
    CHECK FIRST
    Add timeout= and stdin=subprocess.DEVNULL to the call. In a second terminal, ps aux | grep <program> shows whether the child is still running.
  • The script finishes with no error, but the file or deployment it should have produced is missing.

    WHY IT HAPPENS
    A command failed, but the code never looked at returncode, so the script kept going. Another version of this bug is TypeError: a bytes-like object is required, not 'str', which means text=True is missing and the output is bytes.
    CHECK FIRST
    Add check=True and log stderr in the except subprocess.CalledProcessError handler. Add text=True to get strings.

When not to use this

  • Do not shell out for something Python already does. Use pathlib and shutil for files, requests for HTTP, and shutil.disk_usage for disk space. They return real values instead of text you must parse.
  • If the script is only a chain of a few tools joined by pipes, a short Bash script is clearer than building the same thing out of several subprocess calls.
  • subprocess.run waits for the program to finish. For a program that runs for hours or streams output as it goes, you need subprocess.Popen, which is a more advanced tool.

Check yourself

Why is subprocess.run(["echo", user_text]) safer than subprocess.run(f"echo {user_text}", shell=True)?

With a list there is no shell, so characters like ; and | in the text stay plain data instead of starting another command.

What does check=True do?

It raises CalledProcessError when the program exits with a non-zero code, so the failure cannot be ignored by accident.

Which exception means the program was not found, and which means it ran and failed?

FileNotFoundError means it could not be started. CalledProcessError (with check=True) means it started and returned a non-zero exit code.

Lesson 28 of 92 in the DevOps path

Search

Search cheat sheets and projects