Why agents must be async

1. The principle

An agent is mostly waiting. Waiting for the model. Waiting for an API. Waiting for a tool. Waiting for the user.

A synchronous agent blocks the entire process while it waits. One user is fine. One hundred users at once is a queue, and the hundredth user gets a fifteen-second response while the application server holds an OS thread doing nothing but watching a socket. That model does not scale, and worse — it makes patterns like "stream the agent's reasoning while it's still thinking" structurally impossible.

Asynchronous code lets one process supervise hundreds of in-flight requests, each one parked at a await point until its remote call returns. The CPU stays useful. Memory stays small. The user gets responsiveness.

Every capstone in this course is built on asyncio. If async isn't fluent for you, this chapter is the cheapest two hours you'll spend.

2. What "blocking" actually means

Consider a single synchronous LLM call:

import openai

client = openai.OpenAI()

def answer(question: str) -> str:
  response = client.chat.completions.create(
      model="gpt-5",
      messages=[{"role": "user", "content": question}],
  )
  return response.choices[0].message.content

When you call answer(), the thread sits on the socket waiting for OpenAI's reply — typically one to fifteen seconds. During that wait, the thread does nothing. If you serve this from a single-process Flask app, every concurrent user takes a thread, and your process runs out of threads long before it runs out of CPU.

You can fix this with threads. You can fix this with processes. Both have downsides — thread-safety bugs in shared agent state, process startup costs, memory bloat. The idiomatic Python fix in 2026 is asyncio.

3. The async equivalent

import asyncio
import openai

client = openai.AsyncOpenAI()

async def answer(question: str) -> str:
  response = await client.chat.completions.create(
      model="gpt-5",
      messages=[{"role": "user", "content": question}],
  )
  return response.choices[0].message.content


async def main():
  questions = [
      "What is the capital of France?",
      "Explain the speed of light.",
      "Name three colors.",
  ]
  answers = await asyncio.gather(*(answer(q) for q in questions))
  for q, a in zip(questions, answers):
      print(f"Q: {q}\nA: {a}\n")


if __name__ == "__main__":
  asyncio.run(main())

Two things to notice.

First, client = openai.AsyncOpenAI() instead of openai.OpenAI(). Almost every SDK now offers both a sync and an async client. The async client returns coroutines you await; the sync client returns values directly.

Second, asyncio.gather() runs all three questions concurrently. The total latency is max(latencies), not sum(latencies). Three questions that each take two seconds now finish in two seconds total, not six.

That is the whole pitch of async for agents: when you have N concurrent waits, you pay max(N), not sum(N).

4. The vocabulary you need

A few terms get used loosely. Here are the precise definitions.

• Coroutine. A function declared with async def. Calling it returns a coroutine object, not a result. You get the result by await-ing it (or scheduling it).
• Event loop. The single thread that runs all the coroutines, switching between them at await points. asyncio.run() starts one.
• Task. A coroutine that's been handed to the loop to run independently. Created with asyncio.create_task() or implicitly via asyncio.gather().
• Awaitable. Anything you can await: coroutines, tasks, futures. Most SDK calls return awaitables.
• Blocking. Code that holds the event loop. Any synchronous call that takes more than a few milliseconds — time.sleep(), requests.get(), big CPU loops, file I/O on slow disks — is a bug in async code.

The hardest rule for beginners is the last one: never call blocking code from inside the event loop. One synchronous requests.get() in a async def function stalls every other coroutine in the process. We'll come back to this.

5. The mental model for an agent

Picture an agent as a coordinator. It has a queue of waiting work and a queue of in-flight work. When it issues an LLM call, it doesn't sit on the call — it parks the call, yields the event loop, and lets the loop run other coroutines (other users' requests, scheduled jobs, the HTTP server's accept loop). When the LLM call returns, the loop wakes the agent coroutine back up.

That's it. That's the whole pattern. Memorize it.

agent.run()
  ├─ await llm_call()        ◀──── parked. event loop runs other work.
  ├─ resume with reply
  ├─ await tool_call()       ◀──── parked. event loop runs other work.
  ├─ resume with tool result
  └─ return final answer

Every chapter in this course assumes this picture. If it's still fuzzy, stop and read the official asyncio docs before continuing. Forty-five minutes, well spent.

6. Try it

If your answer is "I'd run out of threads / processes / memory / patience" — exactly. That is the problem async solves.

7. What's next

The next unit is the FastAPI skeleton every capstone starts from: a single async service that accepts requests, calls an LLM, and returns answers without blocking. Twenty lines of code, but it's the chassis the rest of the course bolts onto.