Lesson 7 of 7 in Python and scripting for ops

Errors, logging and honest exit codes

Core9 min

An exception is Python's way of signaling that something went wrong, and the logging module is the standard way to record what a script did. Together with a non-zero exit code on failure, they let someone else diagnose a run they were not watching.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Reading and writing files with pathlib, Parsing JSON and YAML config safely

The problem this solves

Ops scripts usually run when nobody is looking: at night from cron, inside a CI job, after a deploy. When one fails, the only evidence is what it left behind. A bare Traceback or, worse, no output at all, makes the next hour miserable.

Using print for everything mixes normal output with diagnostics, gives no timestamps and no severity, and cannot be turned up or down. The two opposite mistakes are catching every error and carrying on silently, and letting every small problem crash the script with an unreadable trace.

A good script says what it was doing, what went wrong, and what to check, then exits with a failing code.

How it works

Where a script's messages and result go
Where a script's messages and result goThe script sends messages through the logging module, which writes them to the console and a log file, and it reports the final outcome as an exit code.1log.info(...)2 handler2 handler3 sys.exit(1)Scriptdoes the worklogginglevel and formatstderrconsole or CI logLog filekept for laterExit codewhat cron and CI read
  • Runs code
  • Control
  • Other
  • Data

The script sends messages through the logging module, which writes them to the console and a log file, and it reports the final outcome as an exit code.

  1. Script to logging: 1 log.info(...)
  2. logging to stderr: 2 handler
  3. logging to Log file: 2 handler
  4. Script to Exit code: 3 sys.exit(1)
What happens when a step fails
What happens when a step failsA failing step raises an exception, the handler logs it with context at the place where the script can act, and the script exits with a non-zero code.1 raise2 caught3 return 1optionalA step failsfile missing, bad JSONExceptiontravels up the callsexcept blocklogs the causeExit code 1tells the callerFull tracebacklog.exception only
  • Runs code
  • Other
  • Control
  • Data

A failing step raises an exception, the handler logs it with context at the place where the script can act, and the script exits with a non-zero code.

  1. A step fails to Exception: 1 raise
  2. Exception to except block: 2 caught
  3. except block to Exit code 1: 3 return 1
  4. except block to Full traceback: optional

An exception travels up through the calling code until something catches it. Catch the specific ones you can explain, such as a missing file or bad JSON, and add context in the message: which file, which line, and what the script was trying to do. Leave unexpected errors uncaught, so the traceback shows up and the exit code is non-zero.

Call logging.basicConfig(level=..., format=...) once at startup. Log with log.info, log.warning and log.error, and use log.exception inside an except block when you want the full traceback in the log. By default logging writes to stderr, so normal output on stdout stays clean.

Log an error once, at the place where you handle it. If every layer logs and re-raises, the same failure shows up four times.

End the script with sys.exit(main()), where main returns 0 on success and a non-zero number on failure. That number is the only thing cron, CI and calling scripts will see.

Try it yourself

Runs in the lab folder with plain python3. You will write one script and run it against a good file, a missing file and a broken file.

  1. 1
    Write a small script that fails well
    cd ~/scripting-lab
    cat > ops.py <<'EOF'
    import json
    import logging
    import os
    import sys
    from pathlib import Path
    
    log = logging.getLogger("ops")
    
    
    def load(path):
        try:
            return json.loads(Path(path).read_text(encoding="utf-8"))
        except OSError as e:
            raise ValueError(f"cannot read {path} (cwd={Path.cwd()}): {e}") from e
        except json.JSONDecodeError as e:
            raise ValueError(f"{path} is not valid JSON, line {e.lineno}: {e.msg}") from e
    
    
    def main():
        logging.basicConfig(
            level=os.environ.get("LOG_LEVEL", "INFO"),
            format="%(asctime)s %(levelname)s %(name)s %(message)s",
        )
        log.debug("argv=%s", sys.argv)
        try:
            cfg = load(sys.argv[1])
        except ValueError as e:
            log.error("%s", e)
            return 1
        log.info("loaded %d keys", len(cfg))
        return 0
    
    
    if __name__ == "__main__":
        sys.exit(main())
    EOF

    load adds context and raises one error type. main logs it once and returns 1. LOG_LEVEL lets you turn detail up without editing the code.

  2. 2
    Run it on a good file
    echo '{"a": 1, "b": 2}' > good.json
    python3 ops.py good.json
    echo "exit=$?"

    You see one INFO line with a timestamp saying it loaded 2 keys, then exit=0.

  3. 3
    Run it on a missing file and a broken file
    python3 ops.py nope.json
    echo "exit=$?"
    echo "{'a': 1}" > bad.json
    python3 ops.py bad.json
    echo "exit=$?"

    Each run prints one ERROR line that says what failed and where, then exit=1. The missing-file message includes the working directory, the first thing you would otherwise have to guess.

  4. 4
    Turn on debug detail and capture logs separately
    LOG_LEVEL=DEBUG python3 ops.py good.json 2> run.log
    cat run.log

    Logging goes to stderr, so 2> run.log captured it. run.log now contains a DEBUG line with the arguments as well as the INFO line.

What breaks, and where to look

  • Your log.info(...) lines never appear, but warnings and errors show up without timestamps.

    WHY IT HAPPENS
    Logging was never configured. The default level only lets warnings and above through, and uses a bare format.
    CHECK FIRST
    Call logging.basicConfig(level=logging.INFO, format=...) once at startup, before any logging. Run python3 -c "import logging; print(logging.getLogger().level)" to see the default level.
  • The cron job "succeeds" every night, but the report is empty and nothing is in any log.

    WHY IT HAPPENS
    An except Exception: pass (or a bare except:) swallowed the error, so the script ended with exit code 0.
    CHECK FIRST
    Search the script with grep -n "except" script.py. Replace silent handlers with log.exception("what failed") and a non-zero return.
  • Every log line appears twice, or three times.

    WHY IT HAPPENS
    Handlers were added more than once, for example by calling basicConfig or addHandler inside a function that runs repeatedly, or a child logger passing messages up to a parent that also prints them.
    CHECK FIRST
    Inspect logging.getLogger("ops").handlers and logging.getLogger().handlers, and configure logging in exactly one place, the start of main.

When not to use this

  • For a ten-line throwaway script, print(..., file=sys.stderr) and a clear SystemExit message are enough. Logging configuration is not worth it until someone else will read the output.
  • Once many scripts and services log, reading files by hand stops scaling. Teams ship logs to a central place and use structured fields. Start with clear messages and the right levels so that later step is easy.

Check yourself

Why must a failing script exit with a non-zero code?

Cron, CI and other scripts only see the exit code. If it is 0, they treat the run as a success.

What is the difference between log.error and log.exception?

log.exception is meant for use inside an except block and also records the full traceback. log.error records only the message.

Why is except Exception: pass dangerous in an ops script?

It hides the failure and lets the script exit 0, so nobody finds out that the job did not do its work.

Lesson 30 of 92 in the DevOps path

Search

Search cheat sheets and projects