Lesson 3 of 7 in Python and scripting for ops
Reading and writing files with pathlib
Files are how a script gets its input and leaves its results. In Python, pathlib.Path represents a file location as an object, with methods to read, write, list and check it.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Bash or Python: picking the right tool
The problem this solves
Ops scripts spend most of their time with text files: logs, configs, lists of hosts, reports. Getting them wrong causes the quietest failures. A script that works from your terminal can fail under cron because it looked for app.log in a different folder.
Encoding is another trap. Python has to turn bytes into text, and if it guesses wrong the script crashes or, worse, produces garbled text.
Writing has its own risk. If your script dies halfway through writing a report, whoever reads it next gets a half-written file.
How it works
- Control
- Data
- Runs code
The script reads a log file, builds a summary, and writes it to a report file, resolving relative paths from the working directory.
- app.log to Script: 1 read_text
- Script to summary.txt: 2 write_text
- Working dir to Script: resolves relative paths
- Runs code
- Data
- People and clients
The script writes the whole report to a temporary file and then replaces the real file in one step, so a reader sees either the old report or the new one.
- Script to summary.tmp: 1 write everything
- summary.tmp to summary.txt: 2 replace in one step
- Reader to summary.txt: 3 reads
Path("app.log").read_text(encoding="utf-8") reads the whole file into one string. .splitlines() turns it into a list of lines. .write_text(...) writes a string out. Always pass encoding="utf-8" so the result does not depend on the machine's default.
A relative path like app.log is resolved from the working directory of whatever started the script. Cron and CI start in a different folder than your terminal. To anchor a path to the script itself, build it from Path(__file__).parent.
For files too big to hold in memory, open the file and loop over it. Python then reads one line at a time.
To avoid half-written output, write to a temporary file in the same folder and call .replace() on it. On the same filesystem the replace happens in one step, so a reader sees the old file or the new one, never a mixture.
Try it yourself
Runs in the lab folder. No packages needed. These steps recreate app.log so they work on their own.
- 1Create the input file
cd ~/scripting-lab cat > app.log <<'EOF' INFO start ERROR disk full INFO retry ERROR disk full ERROR timeout EOF
- 2Write a script that summarizes it
cat > summarize.py <<'EOF' from collections import Counter from pathlib import Path here = Path(__file__).parent lines = (here / "app.log").read_text(encoding="utf-8").splitlines() counts = Counter(line.split(" ", 1)[0] for line in lines if line.strip()) out = here / "summary.txt" tmp = out.with_suffix(".tmp") tmp.write_text("".join(f"{k} {v}\n" for k, v in sorted(counts.items())), encoding="utf-8") tmp.replace(out) print(out.read_text(encoding="utf-8"), end="") EOF python3 summarize.pyYou should see
ERROR 3 INFO 2
The report is written to
summary.tmpfirst and then replaced.Path(__file__).parentis the folder the script lives in. - 3Run it from a different folder
cd / python3 ~/scripting-lab/summarize.py cd ~/scripting-lab
You should see
ERROR 3 INFO 2
It still works because the paths hang off the script's own location, not off where you stood when you ran it.
- 4Read a big file one line at a time
python3 - <<'EOF' from pathlib import Path with Path("app.log").open(encoding="utf-8") as f: n = sum(1 for line in f if line.startswith("ERROR")) print(n) EOFYou should see
3
The loop pulls one line at a time instead of loading the whole file, which keeps memory flat on huge logs.
What breaks, and where to look
FileNotFoundError: [Errno 2] No such file or directory: 'app.log'
- WHY IT HAPPENS
- The path is relative and the script was started from a different working directory, which is typical under cron, systemd or CI.
- CHECK FIRST
- Print
Path.cwd()at the start of the script, then switch to a path built fromPath(__file__).parentor an absolute path from config.
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
- WHY IT HAPPENS
- The file is not UTF-8. Files written with a redirect in Windows PowerShell are often UTF-16, which starts with those bytes.
- CHECK FIRST
- Run
file app.logto see the encoding. Re-save the file as UTF-8, or pass the rightencoding=value. Useerrors="replace"only when losing a few characters is acceptable.
PermissionError: [Errno 13] Permission denied: '/var/log/auth.log'
- WHY IT HAPPENS
- The file belongs to root or a system group and your user is not allowed to read it.
- CHECK FIRST
- Run
ls -l /var/log/auth.logandid. Run the script as a user in the right group instead of reaching forsudo python.
When not to use this
read_text()loads the entire file into memory. For multi-gigabyte logs, loop over the open file or use tools built for the job, likegreporawk.- The temporary-file trick protects readers from half-written files. It does not protect two scripts that write the same file at the same time. For that you need locking or a single writer.
Check yourself
Why pass encoding="utf-8" explicitly?
Without it Python uses the machine's default, so the same script can behave differently on different systems.
Why does a script that works in your terminal fail under cron with FileNotFoundError?
Relative paths start from the working directory, and cron starts in a different one. Anchor paths to the script or use absolute paths.
What does writing to a temp file and calling .replace() buy you?
Readers see the complete old file or the complete new file, never a half-written one.
Lesson 26 of 92 in the DevOps path