Lesson 1 of 7 in Python and scripting for ops

Bash or Python: picking the right tool

Core6 min

Bash is a shell language built for starting programs and connecting them with pipes. Python is a general language built for data, logic and libraries. Ops work needs both, and the skill is knowing which one fits the job in front of you.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Linux and the shell

The problem this solves

Most ops automation starts as a handful of commands you type by hand. You paste them into a file and now you have a script. That works until the script needs to read a config file, loop over a list of servers, or call an API and look at the answer.

Bash can do all of that, but it gets awkward fast. Values are text, errors are easy to ignore, and one unquoted space can change what a command does. Python makes those jobs easier, but it is heavier when all you want is to run three commands in a row.

Picking the wrong tool does not stop you. It just makes the script harder to read and easier to break six months later.

How it works

Choosing a tool by the shape of the job
Choosing a tool by the shape of the jobAn ops task goes to Bash when it is mostly running commands, or to Python when it is mostly data and logic, and both finish by reporting an exit code.1a mostly commands1b mostly data2 exit 0 or not2 exit 0 or notOps tasksomething to automateBashrun and connect commandsPythondata, APIs, decisionsExit codewhat cron and CI read
  • People and clients
  • Runs code
  • Other

An ops task goes to Bash when it is mostly running commands, or to Python when it is mostly data and logic, and both finish by reporting an exit code.

  1. Ops task to Bash: 1a mostly commands
  2. Ops task to Python: 1b mostly data
  3. Bash to Exit code: 2 exit 0 or not
  4. Python to Exit code: 2 exit 0 or not

Look at what the script spends its time doing. If it mostly starts programs and passes their output along, for example grep, sort and tar, Bash is the natural fit. The shell is already the thing that starts programs.

If it parses JSON or YAML, loops with conditions, calls an HTTP API, or needs to explain its own errors clearly, Python is the better fit. It has real data structures and a standard library for files, JSON and logging.

Whichever you pick, the script ends with an exit code. Zero means success. Anything else means failure. Cron, CI systems and other scripts only read that number, so a script that fails but exits zero is lying to everything around it.

A rough rule many teams use is to move from Bash to Python once a script grows past a screen or two, or the first time you want a data structure. It is a judgement call, not a law.

Try it yourself

Do the same small job in both languages. You need a Linux or macOS terminal with python3. Everything stays inside one throwaway folder.

  1. 1
    Make a lab folder and a sample log
    mkdir -p ~/scripting-lab && cd ~/scripting-lab
    cat > app.log <<'EOF'
    INFO start
    ERROR disk full
    INFO retry
    ERROR disk full
    ERROR timeout
    EOF

    All later concepts reuse this folder. Create it once.

  2. 2
    Bash: count the error lines
    grep -c ERROR app.log

    You should see

    3

    One command does the whole job. This is where Bash shines.

  3. 3
    Python: count each distinct error
    python3 - <<'EOF'
    from collections import Counter
    from pathlib import Path
    
    lines = Path("app.log").read_text(encoding="utf-8").splitlines()
    counts = Counter(line.split(" ", 1)[1] for line in lines if line.startswith("ERROR"))
    print(counts.most_common())
    EOF

    You should see

    [('disk full', 2), ('timeout', 1)]

    Grouping by message is a few lines in Python. In Bash it needs a longer pipeline of cut, sort and uniq.

  4. 4
    See that both speak the same exit code language
    python3 -c "import sys; sys.exit(3)"
    echo $?

    You should see

    3

    A Python script can pick its own exit code with sys.exit. The shell sees it the same way as any other program.

What breaks, and where to look

  • bash: python: command not found

    WHY IT HAPPENS
    Many Linux distributions ship only python3, not a command called python.
    CHECK FIRST
    Run command -v python3 and use that name in your scripts and shebang lines.
  • script.sh: 3: [[: not found

    WHY IT HAPPENS
    You ran a Bash script with sh script.sh. On Debian and Ubuntu sh is a smaller shell that does not know Bash features like [[ ]].
    CHECK FIRST
    Look at the first line of the script (head -1 script.sh) and run it with bash script.sh, or make it executable and run ./script.sh.
  • cp: cannot stat 'my': No such file or directory (for a file called my report.txt)

    WHY IT HAPPENS
    The variable was not quoted. Bash split my report.txt into two words and passed them as two arguments.
    CHECK FIRST
    Quote every variable that holds a path: cp "$file" backup/. Run the script with bash -x script.sh to see the exact arguments.

When not to use this

  • Do not rewrite a working five-line Bash script in Python just because Python feels more serious. If it does one thing and nobody has trouble reading it, leave it.
  • Both languages are poor fits for long-running services and heavy data work. Once a script becomes a program with users and tests, treat it like software, not like a one-off.

Check yourself

What kind of job points toward Bash?

A job that is mostly starting programs and piping their output into each other, like filtering a log with grep and sort.

What kind of job points toward Python?

A job that parses JSON or YAML, loops with conditions, calls an API, or needs clear error messages.

Why does the exit code matter whichever language you choose?

Cron, CI and other scripts only read the exit code. A script that fails but exits zero hides the failure from all of them.

Lesson 24 of 92 in the DevOps path

Search

Search cheat sheets and projects