Lesson 4 of 7 in Python and scripting for ops

Parsing JSON and YAML config safely

Core9 min

JSON and YAML are text formats for structured data such as settings, lists and nested objects. Parsing turns that text into Python dictionaries and lists, and validating checks the result has what your script needs.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Python environments: venv and requirements, Reading and writing files with pathlib

The problem this solves

Config files, API responses, CI definitions and Kubernetes manifests are all JSON or YAML. A script that automates any of them has to read them correctly, and a surprising parse can do real damage later, such as deploying the wrong version.

There is also a security side. YAML has a feature that lets a document name Python objects to build. The general loader, yaml.load, honors that, so loading a file from someone else can run code on your machine.

And there is the quiet side. YAML guesses types, so some values do not come out the way you wrote them.

How it works

From text on disk to checked data in your script
From text on disk to checked data in your scriptA config file is read by a parser into plain Python data, and a validation step checks the keys and types before the rest of the script uses it.1 read text2 parse3 checkconfig.yamltext on diskParsersafe_load / json.loaddict and listplain Python dataValidationrequired keys and types
  • Data
  • Control
  • Other
  • Security

A config file is read by a parser into plain Python data, and a validation step checks the keys and types before the rest of the script uses it.

  1. config.yaml to Parser: 1 read text
  2. Parser to dict and list: 2 parse
  3. dict and list to Validation: 3 check
Why safe_load is the loader to use
Why safe_load is the loader to usesafe_load turns YAML into plain dictionaries and lists, while the general yaml.load can build arbitrary Python objects and should be avoided for config.use thisplain dataavoid for configYAML filemaybe not written by yousafe_loadplain data types onlydict and listnumbers, text, booleansyaml.loadcan build any Python object
  • Data
  • Security
  • Other
  • Runs code

safe_load turns YAML into plain dictionaries and lists, while the general yaml.load can build arbitrary Python objects and should be avoided for config.

  1. YAML file to safe_load: use this
  2. safe_load to dict and list: plain data
  3. YAML file to yaml.load: avoid for config

JSON maps directly onto Python: objects become dictionaries, arrays become lists, and true and null become True and None. json.load and json.dumps come with Python, so there is nothing to install.

YAML needs a library, usually PyYAML, installed with pip. Use yaml.safe_load. It only builds plain data, so even a hostile file cannot make your script construct objects or run functions.

PyYAML follows an older YAML version in which the bare words yes, no, on and off are booleans, and a value like 1.10 is the number 1.1. If you mean text, put it in quotes. This catches people with version numbers, country codes and feature flags.

After parsing, check what you got. A short check for required keys, with an error message that names the file and the missing key, saves a long hunt later.

Try it yourself

Activate the .venv from the environments concept first (source .venv/bin/activate) so PyYAML is available. Work in the lab folder.

  1. 1
    Create a config file with some traps in it
    cd ~/scripting-lab
    cat > config.yaml <<'EOF'
    service: web
    replicas: 3
    region: eu
    debug: no
    version: 1.10
    EOF
  2. 2
    Load it safely and look at the types
    python3 -c '
    import yaml
    print(yaml.safe_load(open("config.yaml", encoding="utf-8")))
    '

    You should see

    {'service': 'web', 'replicas': 3, 'region': 'eu', 'debug': False, 'version': 1.1}

    debug: no became False and version: 1.10 became the number 1.1. Edit the file to version: "1.10" and run it again to keep the text.

  3. 3
    Check for required keys with a useful message
    python3 - <<'EOF'
    import yaml
    
    cfg = yaml.safe_load(open("config.yaml", encoding="utf-8"))
    missing = [k for k in ("service", "replicas", "image") if k not in cfg]
    if missing:
        raise SystemExit(f"config.yaml is missing: {', '.join(missing)}")
    EOF
    echo $?

    You should see

    config.yaml is missing: image
    1

    SystemExit with a string prints it to stderr and exits with code 1.

  4. 4
    See what a JSON syntax error looks like
    echo "{'a': 1}" | python3 -m json.tool

    You should see

    Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

    JSON requires double quotes. python3 -m json.tool is also a quick way to validate and pretty-print any JSON file: python3 -m json.tool file.json.

What breaks, and where to look

  • json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

    WHY IT HAPPENS
    The text is not valid JSON. Single quotes and trailing commas are the usual reasons, because JSON is stricter than Python dictionaries or JavaScript.
    CHECK FIRST
    Run python3 -m json.tool file.json. The line and column in the error point at the first bad character.
  • yaml.scanner.ScannerError: found character '\t' that cannot start any token

    WHY IT HAPPENS
    The YAML file is indented with a tab. YAML only allows spaces for indentation.
    CHECK FIRST
    Run grep -nP '\t' config.yaml to find tab characters, and set your editor to insert spaces for YAML files.
  • The deploy uses version 1.1 and a flag is false, but the file says 1.10 and no... as text

    WHY IT HAPPENS
    YAML guessed the types. Unquoted 1.10 became a float and unquoted no became a boolean, with no error.
    CHECK FIRST
    Print type(value) for the suspicious keys after loading, and put quotes around anything that must stay text.

When not to use this

  • PyYAML throws away comments and ordering niceties when you dump a file back out. If a script must edit a hand-maintained YAML file and keep its comments, look at a library built for round-tripping, or avoid editing it from code.
  • Parsing only tells you the text is well formed, not that the values make sense. For larger configs, add real validation with a schema or a library built for it instead of growing a pile of if statements.
  • Do not keep secrets in config files that live in git. Read them from environment variables or a secrets manager.

Check yourself

Why use yaml.safe_load rather than yaml.load?

safe_load only builds plain data, while yaml.load can construct arbitrary Python objects, so a crafted file can run code.

What does debug: no become in PyYAML, and how do you keep it as text?

It becomes the boolean False. Quote it, as in debug: "no", to keep the string.

What is the value of validating right after parsing?

A missing or wrong key fails immediately with a message naming the file and key, instead of causing a confusing failure later.

Lesson 27 of 92 in the DevOps path

Search

Search cheat sheets and projects