Lesson 3 of 7 in Python and scripting for ops

Reading and writing files with pathlib

Core9 min

Files are how a script gets its input and leaves its results. In Python, pathlib.Path represents a file location as an object, with methods to read, write, list and check it.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Bash or Python: picking the right tool

The problem this solves

Ops scripts spend most of their time with text files: logs, configs, lists of hosts, reports. Getting them wrong causes the quietest failures. A script that works from your terminal can fail under cron because it looked for app.log in a different folder.

Encoding is another trap. Python has to turn bytes into text, and if it guesses wrong the script crashes or, worse, produces garbled text.

Writing has its own risk. If your script dies halfway through writing a report, whoever reads it next gets a half-written file.

How it works

A script reading one file and writing another
A script reading one file and writing anotherThe script reads a log file, builds a summary, and writes it to a report file, resolving relative paths from the working directory.1 read_text2 write_textresolves relative pathsWorking dirwhere relative paths startapp.loginput fileScriptpathlib + Countersummary.txtoutput file
  • Control
  • Data
  • Runs code

The script reads a log file, builds a summary, and writes it to a report file, resolving relative paths from the working directory.

  1. app.log to Script: 1 read_text
  2. Script to summary.txt: 2 write_text
  3. Working dir to Script: resolves relative paths
Writing a file so readers never see half of it
Writing a file so readers never see half of itThe script writes the whole report to a temporary file and then replaces the real file in one step, so a reader sees either the old report or the new one.1 writeeverything2 replace inone step3 readsScriptbuilds the reportsummary.tmpscratch copysummary.txtthe real fileReadercron job, person
  • Runs code
  • Data
  • People and clients

The script writes the whole report to a temporary file and then replaces the real file in one step, so a reader sees either the old report or the new one.

  1. Script to summary.tmp: 1 write everything
  2. summary.tmp to summary.txt: 2 replace in one step
  3. Reader to summary.txt: 3 reads

Path("app.log").read_text(encoding="utf-8") reads the whole file into one string. .splitlines() turns it into a list of lines. .write_text(...) writes a string out. Always pass encoding="utf-8" so the result does not depend on the machine's default.

A relative path like app.log is resolved from the working directory of whatever started the script. Cron and CI start in a different folder than your terminal. To anchor a path to the script itself, build it from Path(__file__).parent.

For files too big to hold in memory, open the file and loop over it. Python then reads one line at a time.

To avoid half-written output, write to a temporary file in the same folder and call .replace() on it. On the same filesystem the replace happens in one step, so a reader sees the old file or the new one, never a mixture.

Try it yourself

Runs in the lab folder. No packages needed. These steps recreate app.log so they work on their own.

  1. 1
    Create the input file
    cd ~/scripting-lab
    cat > app.log <<'EOF'
    INFO start
    ERROR disk full
    INFO retry
    ERROR disk full
    ERROR timeout
    EOF
  2. 2
    Write a script that summarizes it
    cat > summarize.py <<'EOF'
    from collections import Counter
    from pathlib import Path
    
    here = Path(__file__).parent
    lines = (here / "app.log").read_text(encoding="utf-8").splitlines()
    counts = Counter(line.split(" ", 1)[0] for line in lines if line.strip())
    
    out = here / "summary.txt"
    tmp = out.with_suffix(".tmp")
    tmp.write_text("".join(f"{k} {v}\n" for k, v in sorted(counts.items())), encoding="utf-8")
    tmp.replace(out)
    print(out.read_text(encoding="utf-8"), end="")
    EOF
    python3 summarize.py

    You should see

    ERROR 3
    INFO 2

    The report is written to summary.tmp first and then replaced. Path(__file__).parent is the folder the script lives in.

  3. 3
    Run it from a different folder
    cd /
    python3 ~/scripting-lab/summarize.py
    cd ~/scripting-lab

    You should see

    ERROR 3
    INFO 2

    It still works because the paths hang off the script's own location, not off where you stood when you ran it.

  4. 4
    Read a big file one line at a time
    python3 - <<'EOF'
    from pathlib import Path
    
    with Path("app.log").open(encoding="utf-8") as f:
        n = sum(1 for line in f if line.startswith("ERROR"))
    print(n)
    EOF

    You should see

    3

    The loop pulls one line at a time instead of loading the whole file, which keeps memory flat on huge logs.

What breaks, and where to look

  • FileNotFoundError: [Errno 2] No such file or directory: 'app.log'

    WHY IT HAPPENS
    The path is relative and the script was started from a different working directory, which is typical under cron, systemd or CI.
    CHECK FIRST
    Print Path.cwd() at the start of the script, then switch to a path built from Path(__file__).parent or an absolute path from config.
  • UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

    WHY IT HAPPENS
    The file is not UTF-8. Files written with a redirect in Windows PowerShell are often UTF-16, which starts with those bytes.
    CHECK FIRST
    Run file app.log to see the encoding. Re-save the file as UTF-8, or pass the right encoding= value. Use errors="replace" only when losing a few characters is acceptable.
  • PermissionError: [Errno 13] Permission denied: '/var/log/auth.log'

    WHY IT HAPPENS
    The file belongs to root or a system group and your user is not allowed to read it.
    CHECK FIRST
    Run ls -l /var/log/auth.log and id. Run the script as a user in the right group instead of reaching for sudo python.

When not to use this

  • read_text() loads the entire file into memory. For multi-gigabyte logs, loop over the open file or use tools built for the job, like grep or awk.
  • The temporary-file trick protects readers from half-written files. It does not protect two scripts that write the same file at the same time. For that you need locking or a single writer.

Check yourself

Why pass encoding="utf-8" explicitly?

Without it Python uses the machine's default, so the same script can behave differently on different systems.

Why does a script that works in your terminal fail under cron with FileNotFoundError?

Relative paths start from the working directory, and cron starts in a different one. Anchor paths to the script or use absolute paths.

What does writing to a temp file and calling .replace() buy you?

Readers see the complete old file or the complete new file, never a half-written one.

Lesson 26 of 92 in the DevOps path

Search

Search cheat sheets and projects