Lesson 7 of 7 in Python and scripting for ops
Errors, logging and honest exit codes
An exception is Python's way of signaling that something went wrong, and the logging module is the standard way to record what a script did. Together with a non-zero exit code on failure, they let someone else diagnose a run they were not watching.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Reading and writing files with pathlib, Parsing JSON and YAML config safely
The problem this solves
Ops scripts usually run when nobody is looking: at night from cron, inside a CI job, after a deploy. When one fails, the only evidence is what it left behind. A bare Traceback or, worse, no output at all, makes the next hour miserable.
Using print for everything mixes normal output with diagnostics, gives no timestamps and no severity, and cannot be turned up or down. The two opposite mistakes are catching every error and carrying on silently, and letting every small problem crash the script with an unreadable trace.
A good script says what it was doing, what went wrong, and what to check, then exits with a failing code.
How it works
- Runs code
- Control
- Other
- Data
The script sends messages through the logging module, which writes them to the console and a log file, and it reports the final outcome as an exit code.
- Script to logging: 1 log.info(...)
- logging to stderr: 2 handler
- logging to Log file: 2 handler
- Script to Exit code: 3 sys.exit(1)
- Runs code
- Other
- Control
- Data
A failing step raises an exception, the handler logs it with context at the place where the script can act, and the script exits with a non-zero code.
- A step fails to Exception: 1 raise
- Exception to except block: 2 caught
- except block to Exit code 1: 3 return 1
- except block to Full traceback: optional
An exception travels up through the calling code until something catches it. Catch the specific ones you can explain, such as a missing file or bad JSON, and add context in the message: which file, which line, and what the script was trying to do. Leave unexpected errors uncaught, so the traceback shows up and the exit code is non-zero.
Call logging.basicConfig(level=..., format=...) once at startup. Log with log.info, log.warning and log.error, and use log.exception inside an except block when you want the full traceback in the log. By default logging writes to stderr, so normal output on stdout stays clean.
Log an error once, at the place where you handle it. If every layer logs and re-raises, the same failure shows up four times.
End the script with sys.exit(main()), where main returns 0 on success and a non-zero number on failure. That number is the only thing cron, CI and calling scripts will see.
Try it yourself
Runs in the lab folder with plain python3. You will write one script and run it against a good file, a missing file and a broken file.
- 1Write a small script that fails well
cd ~/scripting-lab cat > ops.py <<'EOF' import json import logging import os import sys from pathlib import Path log = logging.getLogger("ops") def load(path): try: return json.loads(Path(path).read_text(encoding="utf-8")) except OSError as e: raise ValueError(f"cannot read {path} (cwd={Path.cwd()}): {e}") from e except json.JSONDecodeError as e: raise ValueError(f"{path} is not valid JSON, line {e.lineno}: {e.msg}") from e def main(): logging.basicConfig( level=os.environ.get("LOG_LEVEL", "INFO"), format="%(asctime)s %(levelname)s %(name)s %(message)s", ) log.debug("argv=%s", sys.argv) try: cfg = load(sys.argv[1]) except ValueError as e: log.error("%s", e) return 1 log.info("loaded %d keys", len(cfg)) return 0 if __name__ == "__main__": sys.exit(main()) EOFloadadds context and raises one error type.mainlogs it once and returns 1.LOG_LEVELlets you turn detail up without editing the code. - 2Run it on a good file
echo '{"a": 1, "b": 2}' > good.json python3 ops.py good.json echo "exit=$?"You see one INFO line with a timestamp saying it loaded 2 keys, then
exit=0. - 3Run it on a missing file and a broken file
python3 ops.py nope.json echo "exit=$?" echo "{'a': 1}" > bad.json python3 ops.py bad.json echo "exit=$?"Each run prints one ERROR line that says what failed and where, then
exit=1. The missing-file message includes the working directory, the first thing you would otherwise have to guess. - 4Turn on debug detail and capture logs separately
LOG_LEVEL=DEBUG python3 ops.py good.json 2> run.log cat run.log
Logging goes to stderr, so
2> run.logcaptured it.run.lognow contains a DEBUG line with the arguments as well as the INFO line.
What breaks, and where to look
Your
log.info(...)lines never appear, but warnings and errors show up without timestamps.- WHY IT HAPPENS
- Logging was never configured. The default level only lets warnings and above through, and uses a bare format.
- CHECK FIRST
- Call
logging.basicConfig(level=logging.INFO, format=...)once at startup, before any logging. Runpython3 -c "import logging; print(logging.getLogger().level)"to see the default level.
The cron job "succeeds" every night, but the report is empty and nothing is in any log.
- WHY IT HAPPENS
- An
except Exception: pass(or a bareexcept:) swallowed the error, so the script ended with exit code 0. - CHECK FIRST
- Search the script with
grep -n "except" script.py. Replace silent handlers withlog.exception("what failed")and a non-zero return.
Every log line appears twice, or three times.
- WHY IT HAPPENS
- Handlers were added more than once, for example by calling
basicConfigoraddHandlerinside a function that runs repeatedly, or a child logger passing messages up to a parent that also prints them. - CHECK FIRST
- Inspect
logging.getLogger("ops").handlersandlogging.getLogger().handlers, and configure logging in exactly one place, the start ofmain.
When not to use this
- For a ten-line throwaway script,
print(..., file=sys.stderr)and a clearSystemExitmessage are enough. Logging configuration is not worth it until someone else will read the output. - Once many scripts and services log, reading files by hand stops scaling. Teams ship logs to a central place and use structured fields. Start with clear messages and the right levels so that later step is easy.
Check yourself
Why must a failing script exit with a non-zero code?
Cron, CI and other scripts only see the exit code. If it is 0, they treat the run as a success.
What is the difference between log.error and log.exception?
log.exception is meant for use inside an except block and also records the full traceback. log.error records only the message.
Why is except Exception: pass dangerous in an ops script?
It hides the failure and lets the script exit 0, so nobody finds out that the job did not do its work.
Lesson 30 of 92 in the DevOps path