Lesson 6 of 7 in Python and scripting for ops
Calling HTTP APIs with requests
Most cloud and DevOps tools expose an HTTP API that a script can call to read or change things. The requests library sends the request, and your script must check the status code and handle failures, not just read the body.
Draft. This lesson is written and waiting for a person to run every command and review it. It is safe to learn from, but check the official documentation for anything you depend on.
Read these first: Python environments: venv and requirements, Parsing JSON and YAML config safely
The problem this solves
Anything you can do in a dashboard, you can usually do with an API call: list servers, check a deployment, post a message to a channel. Scripts that call APIs are the glue of ops work.
Networks are unreliable and servers push back. A call can hang, return an error page, say you sent too many requests, or fail once and work a second later. A script that only handles the happy path will hang a pipeline or crash at night.
There is also a credential in almost every real API call. If it is typed into the script, it ends up in git.
How it works
- Security
- Runs code
- Control
- Other
The script reads its token from the environment, sends a request with an authorization header, receives a status and body, and checks the status before using the data.
- API_TOKEN to Script: token, not in code
- Script to API server: 1 GET with header
- API server to Status and JSON: 2 reply
- Status and JSON to Script: 3 check status
- Runs code
- Control
- Other
When a request fails with a temporary error, the script waits a growing amount of time and tries again, and after a few attempts it gives up with a clear error.
- Script to API server: 1 request
- API server to 503 or timeout: 2 temporary error
- 503 or timeout to Wait, retry: 3 sleep
- Wait, retry to Script: 4 try again
- 503 or timeout to Give up: 5 after N tries
requests.get(url, timeout=5) sends a GET request and returns a response object. The status code is r.status_code, and r.json() decodes a JSON body. Credentials usually go in a header, such as Authorization: Bearer <token>, passed with headers={...}.
Always set timeout. By default requests waits forever, so one stuck server can freeze your script. The timeout applies to waiting for bytes from the server, not to the total time of a download.
r.raise_for_status() raises requests.HTTPError for 4xx and 5xx responses. Without it, a 404 or 500 is just a response, and r.json() may even succeed on an error body. Read the token from an environment variable with os.environ["API_TOKEN"], which fails loudly when it is missing.
Some failures are temporary: a timeout, a 429 (too many requests), or a 502, 503 or 504. Retrying those after a short, growing wait often works. A requests.Session with a urllib3 Retry adapter does this for you, while permanent errors like 401 and 404 should fail straight away.
Try it yourself
Use two terminals and no outside service. In the first, you run a tiny local web server. In the second, you call it. Activate the .venv from the environments concept so requests is installed.
- 1Create a fake API and serve it (terminal 1)
cd ~/scripting-lab mkdir -p site echo '{"status": "ok", "version": "1.4"}' > site/health.json cd site python3 -m http.server 8000 --bind 127.0.0.1Leave this running. It serves the files in
site/onhttp://127.0.0.1:8000, reachable only from your own machine. Stop it with Ctrl+C when you finish. - 2Call it and check the answer (terminal 2)
cd ~/scripting-lab source .venv/bin/activate python - <<'EOF' import requests r = requests.get("http://127.0.0.1:8000/health.json", timeout=5) print(r.status_code, r.headers["Content-Type"]) print(r.json()["status"]) EOFYou should see
200 application/json ok
- 3See what a 404 does
python - <<'EOF' import requests r = requests.get("http://127.0.0.1:8000/missing.json", timeout=5) print(r.status_code) try: r.raise_for_status() except requests.HTTPError as e: print(e) EOFYou should see
404 404 Client Error: File not found for url: http://127.0.0.1:8000/missing.json
The 404 did not raise by itself.
raise_for_status()is what turns it into an exception. - 4Send a token from the environment
API_TOKEN=demo-token python - <<'EOF' import os import requests token = os.environ["API_TOKEN"] r = requests.get( "http://127.0.0.1:8000/health.json", headers={"Authorization": f"Bearer {token}"}, timeout=5, ) print(r.request.headers["Authorization"].split()[0], r.status_code) EOFYou should see
Bearer 200
The demo server ignores the header, but a real API would check it. The script prints only the scheme word, never the token itself. Remove
API_TOKEN=demo-tokenfrom the command and the script stops with aKeyError, which is what you want. - 5Retry a temporary failure, then give up
python - <<'EOF' import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry retry = Retry(total=3, backoff_factor=0.5, status_forcelist=[429, 500, 502, 503, 504]) s = requests.Session() s.mount("http://", HTTPAdapter(max_retries=retry)) try: s.get("http://127.0.0.1:9/", timeout=2) except requests.RequestException as e: print("gave up:", type(e).__name__) EOFPort 9 has nothing listening, so every attempt fails. The script waits a growing amount between attempts and then prints one clear line. The exact exception name depends on your system.
What breaks, and where to look
The script just sits there. Ctrl+C shows a traceback ending inside a socket read.
- WHY IT HAPPENS
- No
timeoutwas passed. By defaultrequestswaits indefinitely for the server. - CHECK FIRST
- Add
timeout=5(or another value you choose) to every call, and catchrequests.Timeoutso the script can report which URL was slow.
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: ...
- WHY IT HAPPENS
- The server did not accept the credentials: the token is missing, expired, for the wrong environment, or sent in the wrong header format. An
Authorizationheader is also dropped if a redirect sends you to a different host. - CHECK FIRST
- Print
r.status_code,r.urlandr.headers.get("WWW-Authenticate"), and try the same call withcurl -i -H "Authorization: Bearer $TOKEN" <url>. Never print the token.
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
- WHY IT HAPPENS
- The body is not JSON. It may be an HTML error or login page from a proxy, or an empty body from a 204 response.
- CHECK FIRST
- Before
r.json(), printr.status_code,r.headers.get("Content-Type")andr.text[:200].
When not to use this
- Retrying is only safe for calls that can be repeated without harm. By default the
Retryadapter skips POST for that reason, because retrying a POST can create a second order or send a second message. If you must retry one, use an idempotency key if the API supports them. - For a one-off check in a terminal,
curlis quicker. For big providers such as AWS, use their official SDK rather than raw HTTP, because it handles request signing, pagination and retries for you.
Check yourself
What happens if you omit timeout in requests.get?
The call waits indefinitely, so one stuck server can hang your script or pipeline.
Does a 404 or 500 response make requests.get raise an exception?
No. You get a normal response object, and you must check status_code or call raise_for_status() to turn it into an error.
Which failures are worth retrying and which are not?
Temporary ones such as timeouts, 429, 502, 503 and 504. Permanent ones such as 401 and 404 should fail immediately.
Keep going
Lesson 29 of 92 in the DevOps path