Lesson 6 of 7 in Python and scripting for ops

Calling HTTP APIs with requests

Core11 min

Most cloud and DevOps tools expose an HTTP API that a script can call to read or change things. The requests library sends the request, and your script must check the status code and handle failures, not just read the body.

Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.

Read these first: Python environments: venv and requirements, Parsing JSON and YAML config safely

The problem this solves

Anything you can do in a dashboard, you can usually do with an API call: list servers, check a deployment, post a message to a channel. Scripts that call APIs are the glue of ops work.

Networks are unreliable and servers push back. A call can hang, return an error page, say you sent too many requests, or fail once and work a second later. A script that only handles the happy path will hang a pipeline or crash at night.

There is also a credential in almost every real API call. If it is typed into the script, it ends up in git.

How it works

One API call, from token to checked answer
One API call, from token to checked answerThe script reads its token from the environment, sends a request with an authorization header, receives a status and body, and checks the status before using the data.token, not in code1 GET with header2 reply3 check statusAPI_TOKENenvironment variableScriptrequests.getAPI serverthe other sideStatus and JSON200, 404, 429, 500
  • Security
  • Runs code
  • Control
  • Other

The script reads its token from the environment, sends a request with an authorization header, receives a status and body, and checks the status before using the data.

  1. API_TOKEN to Script: token, not in code
  2. Script to API server: 1 GET with header
  3. API server to Status and JSON: 2 reply
  4. Status and JSON to Script: 3 check status
What a retry does when the server stumbles
What a retry does when the server stumblesWhen a request fails with a temporary error, the script waits a growing amount of time and tries again, and after a few attempts it gives up with a clear error.1 request2 temporary error3 sleep4 try again5 after NtriesScriptsends the requestAPI serverbusy or restarting503 or timeouttemporary errorWait, retrybackoff grows each timeGive upraise a clear error
  • Runs code
  • Control
  • Other

When a request fails with a temporary error, the script waits a growing amount of time and tries again, and after a few attempts it gives up with a clear error.

  1. Script to API server: 1 request
  2. API server to 503 or timeout: 2 temporary error
  3. 503 or timeout to Wait, retry: 3 sleep
  4. Wait, retry to Script: 4 try again
  5. 503 or timeout to Give up: 5 after N tries

requests.get(url, timeout=5) sends a GET request and returns a response object. The status code is r.status_code, and r.json() decodes a JSON body. Credentials usually go in a header, such as Authorization: Bearer <token>, passed with headers={...}.

Always set timeout. By default requests waits forever, so one stuck server can freeze your script. The timeout applies to waiting for bytes from the server, not to the total time of a download.

r.raise_for_status() raises requests.HTTPError for 4xx and 5xx responses. Without it, a 404 or 500 is just a response, and r.json() may even succeed on an error body. Read the token from an environment variable with os.environ["API_TOKEN"], which fails loudly when it is missing.

Some failures are temporary: a timeout, a 429 (too many requests), or a 502, 503 or 504. Retrying those after a short, growing wait often works. A requests.Session with a urllib3 Retry adapter does this for you, while permanent errors like 401 and 404 should fail straight away.

Try it yourself

Use two terminals and no outside service. In the first, you run a tiny local web server. In the second, you call it. Activate the .venv from the environments concept so requests is installed.

  1. 1
    Create a fake API and serve it (terminal 1)
    cd ~/scripting-lab
    mkdir -p site
    echo '{"status": "ok", "version": "1.4"}' > site/health.json
    cd site
    python3 -m http.server 8000 --bind 127.0.0.1

    Leave this running. It serves the files in site/ on http://127.0.0.1:8000, reachable only from your own machine. Stop it with Ctrl+C when you finish.

  2. 2
    Call it and check the answer (terminal 2)
    cd ~/scripting-lab
    source .venv/bin/activate
    python - <<'EOF'
    import requests
    
    r = requests.get("http://127.0.0.1:8000/health.json", timeout=5)
    print(r.status_code, r.headers["Content-Type"])
    print(r.json()["status"])
    EOF

    You should see

    200 application/json
    ok
  3. 3
    See what a 404 does
    python - <<'EOF'
    import requests
    
    r = requests.get("http://127.0.0.1:8000/missing.json", timeout=5)
    print(r.status_code)
    try:
        r.raise_for_status()
    except requests.HTTPError as e:
        print(e)
    EOF

    You should see

    404
    404 Client Error: File not found for url: http://127.0.0.1:8000/missing.json

    The 404 did not raise by itself. raise_for_status() is what turns it into an exception.

  4. 4
    Send a token from the environment
    API_TOKEN=demo-token python - <<'EOF'
    import os
    import requests
    
    token = os.environ["API_TOKEN"]
    r = requests.get(
        "http://127.0.0.1:8000/health.json",
        headers={"Authorization": f"Bearer {token}"},
        timeout=5,
    )
    print(r.request.headers["Authorization"].split()[0], r.status_code)
    EOF

    You should see

    Bearer 200

    The demo server ignores the header, but a real API would check it. The script prints only the scheme word, never the token itself. Remove API_TOKEN=demo-token from the command and the script stops with a KeyError, which is what you want.

  5. 5
    Retry a temporary failure, then give up
    python - <<'EOF'
    import requests
    from requests.adapters import HTTPAdapter
    from urllib3.util.retry import Retry
    
    retry = Retry(total=3, backoff_factor=0.5, status_forcelist=[429, 500, 502, 503, 504])
    s = requests.Session()
    s.mount("http://", HTTPAdapter(max_retries=retry))
    try:
        s.get("http://127.0.0.1:9/", timeout=2)
    except requests.RequestException as e:
        print("gave up:", type(e).__name__)
    EOF

    Port 9 has nothing listening, so every attempt fails. The script waits a growing amount between attempts and then prints one clear line. The exact exception name depends on your system.

What breaks, and where to look

  • The script just sits there. Ctrl+C shows a traceback ending inside a socket read.

    WHY IT HAPPENS
    No timeout was passed. By default requests waits indefinitely for the server.
    CHECK FIRST
    Add timeout=5 (or another value you choose) to every call, and catch requests.Timeout so the script can report which URL was slow.
  • requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: ...

    WHY IT HAPPENS
    The server did not accept the credentials: the token is missing, expired, for the wrong environment, or sent in the wrong header format. An Authorization header is also dropped if a redirect sends you to a different host.
    CHECK FIRST
    Print r.status_code, r.url and r.headers.get("WWW-Authenticate"), and try the same call with curl -i -H "Authorization: Bearer $TOKEN" <url>. Never print the token.
  • requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

    WHY IT HAPPENS
    The body is not JSON. It may be an HTML error or login page from a proxy, or an empty body from a 204 response.
    CHECK FIRST
    Before r.json(), print r.status_code, r.headers.get("Content-Type") and r.text[:200].

When not to use this

  • Retrying is only safe for calls that can be repeated without harm. By default the Retry adapter skips POST for that reason, because retrying a POST can create a second order or send a second message. If you must retry one, use an idempotency key if the API supports them.
  • For a one-off check in a terminal, curl is quicker. For big providers such as AWS, use their official SDK rather than raw HTTP, because it handles request signing, pagination and retries for you.

Check yourself

What happens if you omit timeout in requests.get?

The call waits indefinitely, so one stuck server can hang your script or pipeline.

Does a 404 or 500 response make requests.get raise an exception?

No. You get a normal response object, and you must check status_code or call raise_for_status() to turn it into an error.

Which failures are worth retrying and which are not?

Temporary ones such as timeouts, 429, 502, 503 and 504. Permanent ones such as 401 and 404 should fail immediately.

Lesson 29 of 92 in the DevOps path

Search

Search cheat sheets and projects