Python
Intermediate 14 min read Python

Async Python for AI

Concurrency without threads for I/O-bound work

LLM calls, vector lookups and web fetches are all I/O-bound. Async Python runs thousands of them concurrently on one thread — the difference between minutes and seconds in a batch embedding pipeline.

asyncioawaitconcurrencyhttpxsemaphorestimeouts
Prerequisites: Data Structures for AI Work
Before we start

What you will learn

  • Explain how async I/O achieves concurrency on one thread
  • Apply asyncio.gather and semaphores to batch model calls
  • Add timeouts and retries that respect concurrency limits
Introduction

What is this?

Async Python (`asyncio`) runs I/O-bound code concurrently on a single thread using an event loop. When a coroutine awaits an I/O operation, the loop schedules other coroutines until the I/O completes. For AI work, where latency hides behind network calls to model and vector APIs, async lets you fire many requests and collect results quickly — with careful concurrency limits.

Motivation

Why do you need it?

  • Embedding 100k documents synchronously can take hours of wall time; async batches cut it to minutes.
  • A single awaiting thread scales far better than hundreds of OS threads for network-bound work.
  • Timeouts and retries compose naturally with `asyncio.timeout` and gather's error handling.
  • Most AI SDKs expose async clients (httpx-based), so the tooling is already async-first.
Think about it

Mental model

Imagine

A chef juggling many orders

Think of

An asyncio event loop

The chef doesn't cook each order start-to-finish; they kick a pot off, move to the next prep while it boils, and return when it bubbles. The event loop does the same with I/O: progress on many requests interleaved, not parallelism on one.

Mechanics

How it works

  • A coroutine (async def) awaits I/O; the event loop parks it and runs other coroutines during the wait.
  • `asyncio.gather` runs several coroutines concurrently and collects their results or first exceptions.
  • A semaphore caps how many in-flight requests exist, protecting provider rate limits.
  • `asyncio.timeout` (py3.11+ / anyio) bounds how long a group may wait, preventing hangs.
  • Results return in order with gather, or as they complete with asyncio.as_completed.

Interleaved awaits on one thread.

Request A
Request B
Request C
A completes
B, C complete
Hands-on

Examples

Bounded-concurrency embedding fan-out

Bounded-concurrency embedding fan-out
import asyncio, random
 
async def embed(chunk_id: str) -> str:
    await asyncio.sleep(0.01 * random.randint(1, 5))  # pretend model call
    return f"vec({chunk_id})"
 
async def main(ids):
    sem = asyncio.Semaphore(5)          # max 5 in flight
    async def one(i):
        async with sem:
            return await embed(i)
    return await asyncio.gather(*[one(i) for i in ids], return_exceptions=True)
 
asyncio.run(main([f"c{i}" for i in range(20)]))

Twenty embeds run with at most five in flight. `asyncio.gather` collects all results or surfaces exceptions instead of killing the batch.

Real world

In production

  • A backfill job embeds 50k documents with a semaphore of 20 concurrent calls, respecting the provider's rate limit and finishing in minutes instead of hours.
  • An agent loop awaits two independent tools (web search + vector search) concurrently, cutting perceived latency in half.
  • A gateway proxies many model providers with per-provider semaphores so one slow upstream doesn't exhaust the pool.
Guidelines

Best practices

  • Always bound concurrency with a semaphore or a pool; unbounded gather hammers rate limits.
  • Wrap every external call in a timeout — a missing timeout is the classic async hang.
  • Use `return_exceptions=True` in gather and handle failures per-item.
  • Keep I/O in the async layer and move heavy CPU work (decoding, parsing) to workers or off the loop.
Watch out

Common mistakes

  • Calling blocking (`requests`, sync DB) inside async and stalling the whole loop.
  • No semaphore — firing 10,000 simultaneous embedding requests and getting 429s.
  • Passing one coroutine per gather but forgetting the list unpack, so they never start.
  • Ignoring exceptions in gather and watching a batch half-complete silently.
Root cause analysis

Common issues

Error
Rate limit 429s during a gather batch
Root cause

Blasting the provider from one process with no concurrency cap

Fix

Add a semaphore sized to the provider's documented limit and retry with backoff.

Error
Whole event loop appears frozen
Root cause

A synchronous library blocks the loop, serializing all coroutines

Fix

Route blocking work to a thread (`asyncio.to_thread`) or swap to the async client.

Error
One failed embed makes the entire batch fail
Root cause

gather propagated the first exception and cancelled the rest

Fix

Use return_exceptions=True, then filter and retry the failures.

When things break

Debugging tips

  • A single slow coroutine making the whole batch slow usually means a blocking call in the loop or a shared semaphore held too long.
  • Timeouts that fire spuriously often come from a sync call hidden behind an async wrapper.
  • Use `asyncio.CancelledError` handling to clean up partial batches on cancellation.
  • Log in-flight counts (semaphore value) to prove concurrency is actually bounded.
Interviews

Interview checkpoint

Practice

Hands-on lab

Parallel Embedding-Batch Emulator
25 min · intermediate
In progress
Objectives
  • Fan out many simulated model calls concurrently with gather
  • Bounds in-flight requests with a semaphore
  • Add a timeout and handle partial failures gracefully

Parallel Embedding-Batch Emulator

25 min · intermediate

0/4 steps
0%
sandbox · Linux sandbox
$
Try these commands
Test yourself

Quick quiz

0/2
1

What happens when an async coroutine awaits I/O?

2

A semaphore is used to bound how many requests may be in flight.

Go deeper

Official documentation

asyncio documentation
docs
httpx — async HTTP client
docs