Chapter 9 · Production serving

APIs and API Design

Every chapter so far ended at a model, a vector index, or an inference engine. This one builds the layer a real frontend actually talks to — a React app calling a backend where some endpoints return JSON in milliseconds and others stream tokens from a GPU for thirty seconds. Designing for both, in the same API, is the chapter.

23 sections FastAPI 0.141 · Pydantic 2.13 React + LLM backend 10 interview drills Reading time ~2.5 hours

[!] The concrete scenario this chapter designs for

A React single-page app is the only client. It needs ordinary CRUD-shaped endpoints — list conversations, save a document, fetch a user's settings — that behave like every REST API you've used. It also needs at least one endpoint that triggers an LLM generation call from chapter 6's vLLM fleet or chapter 8's agent loop, which does not fit the request/response shape those CRUD endpoints use at all.

That tension — one API surface, two fundamentally different response shapes — is why this chapter exists instead of just linking a framework's quickstart. Everything in Part E is downstream of it.

9.1 What this backend actually has to do

A React app cannot talk to a database, a vector index, or a vLLM fleet directly, and should not even if it technically could. The API is the thing in between, and its job is narrower than it sounds: turn HTTP requests into calls against your actual systems, and turn the results back into HTTP responses the browser can use.

[def] Why a browser can't just call your database or your model

A React app runs entirely in a stranger's browser. It cannot hold a database credential, because opening dev tools reveals every string in the page's JavaScript. It cannot connect directly to a vLLM replica on an internal network the browser has no route to. The API is the only thing that runs somewhere you control, so it is the only place credentials, business rules, and internal network access can safely live.

Concretely, for the scenario this chapter designs against, the API sits in the middle of three different kinds of traffic:

What the React app needs, and what actually serves it
The frontend asks for	What actually handles it	Response shape
Load a user's saved conversations	A database query	One JSON object, fast, done
Search past documents by meaning	The vector index from chapter 4	One JSON array, fast, done
Ask a question and get an answer	Chapter 5's RAG pipeline, chapter 6's vLLM fleet	Tokens arriving over seconds — not one object
Run a multi-step agent task	Chapter 8's tool-calling loop	Progress updates over tens of seconds, then a result

[+] The API is not "the backend" — it is a translation layer

Nothing in this chapter asks you to build a database, a search index, or an inference engine again; chapters 4, 5, 6, and 8 already did that. The API's entire job is to sit in front of those systems and speak two languages: HTTP to the browser, and whatever each backend system actually speaks internally. Getting that translation layer right is a smaller problem than it looks, and a more consequential one to get wrong, because every client goes through it.

[retail] Why "API design" is worth a whole chapter, not a paragraph

Most of what makes an API pleasant or miserable to build a frontend against has nothing to do with which framework generated it. It's contract stability, honest error messages, and endpoints that behave the way their HTTP verb implies. A badly designed API built with the trendiest framework is still a badly designed API. This chapter spends its first half on those framework-agnostic decisions before it ever mentions FastAPI by name.

9.2 HTTP, the parts that matter for an API

You have used HTTP as a client your entire career. Designing an API means knowing the handful of pieces the browser and the framework both take seriously, because getting one of them wrong produces a bug that looks like it's somewhere else entirely.

The methods an API actually needs, and what each one promises
Method	Meant for	Promise it makes
`GET`	Fetching a resource	Safe: never changes state. A browser, proxy, or React effect can call it speculatively without side effects.
`POST`	Creating something, or any action that isn't a clean fit for the others	Neither safe nor idempotent by default — calling it twice may create two of something.
`PUT`	Replacing a resource wholesale	Idempotent: calling it 5 times with the same body leaves the same end state as calling it once.
`PATCH`	Partially updating a resource	Not guaranteed idempotent — "increment this counter" is a valid PATCH and isn't.
`DELETE`	Removing a resource	Idempotent by convention: deleting something already gone should still report success.

[!] "Idempotent" is a promise you keep in code, not one HTTP enforces

Nothing stops you from writing a PUT handler that appends to a list instead of replacing it. The method name is a contract with every client, proxy, and retry mechanism between the browser and your server — React's fetch layer, an API gateway, a mobile client rebuilt later — all of which may silently retry a PUT or DELETE on a flaky connection because the method promises that's safe. Break the promise and those retries corrupt data instead of doing nothing.

[def] Status codes: the part actually worth memorising

200 / 201 / 204: OK, Created, No Content. The three success codes worth distinguishing: 201 for a POST that made something new (with a Location header pointing at it), 204 for a DELETE with nothing to return.
400 vs. 422: 400 Bad Request is malformed syntax the server can't even parse. 422 Unprocessable Entity is syntactically valid JSON that fails validation — a missing required field, a string where a number belongs. FastAPI uses 422 for this; know the distinction even if a framework picks one for you.
401 vs. 403: 401 Unauthorized means "I don't know who you are" — no valid credentials presented. 403 Forbidden means "I know exactly who you are, and you may not do this." Returning 401 for a permissions failure tells an attacker to try harder at authenticating when the real problem is authorization.
429: Too Many Requests. The correct response to a client exceeding a rate limit — covered properly in 9.22 — and it should carry a Retry-After header telling the client when to try again.
500 vs. 503: 500 Internal Server Error is your code's bug. 503 Service Unavailable is "I'm fine, but something I depend on isn't" — the right code when the vLLM fleet behind your endpoint is unreachable, distinct from your API itself being broken.

[retail] Statelessness is the property that makes horizontal scaling free

HTTP is stateless by design: each request carries everything the server needs to handle it, with no memory of prior requests required. That is precisely what makes it possible to run ten identical copies of your API behind a load balancer — the exact pattern chapters 6 and 7 relied on for the inference layer — and have any of them handle any request. The instant you store per-user state in a server process's memory instead of a database or a token, you've broken that property, and scaling out silently becomes scaling out with sticky sessions, a much worse deal.

9.3 Designing resources and URLs

REST's actual contribution, stripped of the debate around it, is a naming convention: URLs name things, not actions, and the HTTP method supplies the verb.

RPC-shaped: verbs in the URL

POST /getUser?id=42
POST /createConversation
POST /deleteMessage?id=9

Every endpoint is its own method name. Nothing about the URL tells you whether it's safe to retry, and the list of endpoints grows with every new action.

Resource-shaped: nouns in the URL

GET    /users/42
POST   /conversations
DELETE /messages/9

The URL names a thing; the method says what to do to it. A new client that knows only the noun and the HTTP verb conventions from 9.2 can often guess the rest correctly without reading documentation.

[+] A convention worth adopting even where REST purists disagree

Plural nouns for collections (/conversations), the singular resource nested under it for a specific item (/conversations/{id}), and nested resources for genuine ownership (/conversations/{id}/messages). Consistency matters more than which exact convention you pick — the entire value is a frontend developer guessing an endpoint's shape correctly before checking.

[!] Not everything is a resource, and forcing it usually makes things worse

"Generate an answer to this question" is not really creating, reading, updating, or deleting anything — it's an action. Contorting it into POST /generations to satisfy REST purism is fine as long as it doesn't obscure what's actually happening. A pragmatic action-shaped endpoint like POST /chat/completions — the shape both OpenAI's API and most inference servers settled on — is a more honest name than forcing a resource noun onto something that creates nothing durable. Know the convention, but don't let it override clarity.

[retail] Design the URL for the client's mental model, not your database schema

/conversations/{id}/messages should exist because a React component thinks "give me this conversation's messages," not because your database happens to have a messages table with a foreign key. If your internal schema changes — messages move to a separate service, get split across two tables — the URL shouldn't have to, because nothing about the client's mental model changed. This is the same information-hiding argument as an interface in any other kind of software: the URL is the interface; the schema is the implementation behind it.

9.4 Modeling requests and responses

Every endpoint needs an answer to two questions before a line of handler code exists: exactly what shape comes in, and exactly what shape goes out. Deciding this upfront, as a schema, is what lets a frontend and backend team build in parallel against an agreed contract instead of discovering mismatches at integration time.

[def] Path params, query params, and body: three places data can live

Path parameters: Part of the URL itself, identifying a specific resource: the 42 in /users/42. Required by definition — there is no such thing as an optional path parameter.
Query parameters: The ?key=value pairs after a URL, for filtering, sorting, and pagination on GET requests: /conversations?limit=20&after=cur_9.
Request body: A JSON payload sent with POST, PUT, or PATCH, for data too structured or too large to put in a URL. Never used with GET — some clients and proxies silently drop a GET body.

[+] The rule that resolves most "where does this field go" arguments

If it identifies which resource, it's a path parameter. If it filters or shapes how much of a resource comes back, it's a query parameter. If it's the resource's actual content being created or replaced, it's the body. A search query string is a query parameter even on a complex search endpoint, because it's shaping the response, not identifying a specific existing resource.

[!] A response envelope is a decision, not a default

Returning a bare array from a list endpoint — [{...}, {...}] — means you can never add pagination metadata later without a breaking change, because there's no object to hang a total_count or next_cursor field onto. Wrapping every list response in {"data": [...], "meta": {...}} from day one costs nothing when you don't need meta yet, and avoids a breaking change later when you do — the same lesson chapter 5 taught about designing a database schema for the columns you'll need, not the ones you need today.

A response envelope with room to growjson

{
  "data": [
    {"id": "conv_1", "title": "Q3 planning", "updated_at": "2026-08-10T12:00:00Z"}
  ],
  "meta": {
    "total_count": 143,
    "next_cursor": "cur_9f3a"
  }
}

9.5 Validation: client error vs. server bug

Every field in every request needs an answer to one question: what happens when it's missing, the wrong type, or out of range? Answering it once, at the boundary, is what keeps that question from being re-litigated inside every handler.

[def] Validate at the edge, trust everything after it

The moment a request passes validation, every function it touches afterward should be able to assume its inputs are well-formed — the right types, required fields present, values within declared ranges. Re-checking "is this actually a string" three layers deep into business logic is a sign validation didn't happen where it should have. This is the same boundary-drawing instinct as chapter 8's tool schemas: define the contract once, at the entry point, and let everything behind it rely on it holding.

Validation scattered through handler code

def create_message(body: dict):
    if "content" not in body:
        raise HTTPException(400, "missing content")
    if not isinstance(body["content"], str):
        raise HTTPException(400, "content must be a string")
    if len(body["content"]) > 10_000:
        raise HTTPException(400, "content too long")
    # ...actual logic starts here, finally

Validation declared once, as a schema

class CreateMessage(BaseModel):
    content: str = Field(max_length=10_000)

def create_message(body: CreateMessage):
    # body.content is guaranteed present, str, <= 10,000 chars.
    # Actual logic starts on line one.

[retail] Why this distinction matters more once an LLM is in the loop

A request that passes validation but contains a prompt injection attempt, an absurdly long conversation history that will blow the context window, or a temperature value technically in range but nonsensical for the use case is not a validation failure — validation only checks shape, not meaning. Chapter 8's tool input validation and this section's request validation are the same idea at different layers: both catch malformed input before it reaches code that assumes well-formed input, and neither one is a substitute for the other.

9.6 Designing errors on purpose

An error response is still part of the contract. A frontend has to parse it, decide whether to retry, and decide what to show the user — all of which is impossible if every endpoint's errors look different.

[!] A stack trace is not an error response

Returning a raw exception message and traceback to the client leaks internal structure — file paths, library versions, sometimes fragments of a query — and gives a frontend nothing structured to act on. It is also a real security exposure: a stack trace can reveal exactly which library version to target with a known vulnerability.

One error shape, used everywherejson

{
  "error": {
    "code": "validation_error",
    "message": "content exceeds maximum length of 10000 characters",
    "field": "content"
  }
}

[+] A machine-readable code, a human-readable message

code is what your React app's error handling switches on — stable, never changes wording, safe to write if (error.code === "rate_limited") against. message is what a developer reads in a log or, sometimes, what gets shown to a user. Conflating the two — matching on the human message string — breaks the frontend the moment someone rewords an error for clarity.

[retail] LLM-call errors need a code a human product decision can hang off

A generation call can fail for reasons a database call never does: the model refused on a safety ground, the context window was exceeded, the upstream vLLM fleet from chapter 6 is out of capacity and returning 503s. Each of those deserves a distinct code — content_filtered, context_length_exceeded, model_unavailable — because the right React behaviour is different for each: show a message, trim history and retry, or back off and retry later. Collapsing them all into a generic generation_failed forces the frontend to guess.

9.7 Evolving a contract someone already depends on

The moment a React app is deployed against an endpoint, that endpoint has a client who cannot be forced to update instantly. Three ordinary decisions — how a list paginates, how a list is filtered, and how the whole API is versioned — all come back to the same constraint: changing a contract in place breaks whoever's still on it.

[def] Cursor pagination over offset pagination, for anything that changes

?page=3&limit=20 looks simple until rows are inserted between requests: page 3 silently shifts, and a user scrolling a live feed sees duplicates or skips. A cursor — an opaque token pointing at "after this specific row" — is stable under inserts because it doesn't depend on position at all, only on identity. Offset pagination is fine for a static or rarely-changing table; a conversation list that grows while a user has the page open needs a cursor.

Cursor-based pagination, unaffected by insertsjson

GET /conversations?limit=20&after=cur_9f3a

{
  "data": [ /* 20 conversations */ ],
  "meta": {"next_cursor": "cur_a812", "has_more": true}
}

[+] Filtering and sorting as query parameters, not new endpoints

GET /conversations?status=archived&sort=-updated_at is one endpoint that can answer many questions, rather than /archivedConversations, /conversationsSortedByDate, and every other combination growing into its own route. This is the same resource-not-verb thinking from 9.3, applied to the query string instead of the path.

[!] Version the contract before you need to break it, not after

/v1/conversations costs nothing to add on day one and becomes the only way to change a response shape later without breaking every client still on the old version. The alternative — adding a version prefix retroactively, once v2 is actually needed — means every existing client's URLs are now ambiguously "v1 by default," which is a worse migration than just starting versioned. Header-based versioning (Accept: application/vnd.api+json;version=2) is the more "correct" REST answer and genuinely better for gradual rollouts, but a URL prefix is dramatically easier for a frontend team to reason about and debug, and that's usually the deciding factor for an internal API with one client.

9.8 What's actually out there

Before FastAPI gets any credit, it's worth seeing the field it's competing in. Every framework below can build the API this chapter designs; the differences are in what comes built in versus what you assemble yourself.

Versions verified against the package index, August 2026 (Python) — others by general market knowledge
Framework	Language	Core model	Known for
Flask 3.1.3	Python	Minimal, unopinionated microframework	Simplicity; validation, docs, and async are all add-on choices you make yourself.
Django 6.1 + DRF 3.18.0	Python	Full-stack framework plus a REST toolkit on top	Batteries included: ORM, admin panel, auth, migrations. Heavier than an API-only backend usually needs.
FastAPI 0.141.1	Python	Type hints as the schema	Validation, docs, and serialization all generated from the same type annotations. Detailed in 9.9.
Express	Node.js	Minimal middleware chain	The Flask of Node — ubiquitous, unopinionated, everything else is a library choice.
NestJS	Node.js (TypeScript)	Decorator-based, dependency-injected, Angular-inspired	Closest Node equivalent to FastAPI's structure — a natural fit if the team already writes TypeScript for the React frontend.
Spring Boot	Java / Kotlin	Full dependency-injection container	Enterprise-grade tooling and stability; a heavier runtime and slower iteration loop than a Python or Node API.
Gin / Fiber	Go	Minimal router, compiled binary	Raw throughput and a tiny memory footprint; validation and docs are manual or third-party.

[!] The Node and JVM rows above are not wheel-verified like the rest of this course

Every other chapter in this course pulled real package versions off an artifact mirror before writing about them. This course's tooling only reaches the Python package index, so the Express, NestJS, Spring Boot, and Go rows reflect general, widely known facts about those ecosystems rather than a version-pinned inspection. Treat them as directionally correct, and verify current versions yourself before quoting them in an interview.

[+] The question that actually decides between them

Not "which is fastest" — at the request volumes most APIs actually see, the framework is rarely the bottleneck; the database or the LLM call is. The real question is what you want generated for you versus assembled by hand: request validation, interactive documentation, and dependency injection are either free or bolted on, depending which row of the table you're in. Chapter 7's framework-landscape section made the same point about agent frameworks — the mechanism is available everywhere; what differs is what comes packaged.

9.9 Why FastAPI won this stack

For the specific pairing this chapter designs for — a typed React frontend, a team that wants interactive API docs for free, and at least one endpoint that streams an LLM response — FastAPI's defaults line up with the actual requirements more closely than any competitor's.

[def] The single idea underneath all of it

A FastAPI path operation's type hints are not decoration — they are read at import time and used to build a Pydantic validator, generate the OpenAPI schema, and serialize the response, all from the one annotation you already had to write for your editor's autocomplete. Every feature below is a consequence of that single decision, not a separate feature bolted on.

What one type hint buys you
You write	FastAPI derives
async def get_user(user_id: int)	Path parameter validated as an integer; a non-numeric ID returns 422 automatically, before your function runs.
body: CreateMessage (a Pydantic model)	Full request body validation, an OpenAPI schema for that request, and IDE autocomplete on body.content.
-> MessageOut return annotation	Response serialization and a documented, typed response schema a frontend team can generate a TypeScript client from — 9.14 covers this directly.

[+] Native SSE support, as of version 0.141.1

This is worth calling out because it's genuinely new: FastAPI now ships its own fastapi.sse module with an EventSourceResponse and a ServerSentEvent Pydantic model, built against the OpenAPI 3.2 spec's own section on server-sent events. Streaming an LLM's tokens to a browser — this chapter's Part E problem — used to require a third-party package (sse-starlette); as of this version it's in the framework itself. 9.18 builds a real endpoint with it.

[retail] Async by default fits an I/O-bound backend, not a compute-bound one

Every endpoint in the scenario this chapter designs for — a database query, a vector search, an HTTP call to a vLLM fleet — spends nearly all its time waiting on something else, exactly the I/O-bound pattern chapter 7 built Ray actors around. FastAPI's async support means one worker process can hold hundreds of these requests concurrently without blocking, the same underlying reason chapter 6's vLLM engine batches continuously instead of serving requests one at a time. A CPU-bound API doing heavy in-process computation would not see the same benefit — 9.13 covers exactly where that boundary is.

9.10 What FastAPI still doesn't solve for you

Picking a favourite tool is not the same as pretending it has no edges. FastAPI is deliberately a web layer, not a full-stack framework, and several things Django ships by default are choices you still have to make yourself.

Gaps worth knowing about before they surprise you
Missing piece	What fills it
ORM	None bundled. SQLAlchemy or SQLModel are the common choices — a real decision, not a default.
Admin panel	Nothing built in, unlike Django's automatic admin UI over any model. Usually not missed in an API-only backend with no server-rendered pages, but worth knowing it's not there if the team expects it.
Background job queue at scale	BackgroundTasks (9.19) is fire-and-forget within one process. A durable, retryable, distributed job queue still means Celery, RQ, or arq — the same "this scales to one process, not a fleet" gap chapter 7 flagged for a Ray ActorPool at startup.
Batteries-included auth	Django REST Framework ships permission classes and session auth out of the box; FastAPI gives you the dependency-injection primitives (9.12) to build auth, not auth itself. 9.21 builds it from those primitives.

[!] "Unopinionated" cuts both ways

The freedom to choose your own ORM and auth library is also the freedom for two engineers on the same team to choose differently, with nothing in the framework stopping them. Django's opinions are a cost when they don't fit and a benefit when they do — the same trade-off chapter 7 raised about a fully custom Ray orchestration layer versus an opinionated agent framework. Know which trade you're making, rather than assuming "more flexible" is strictly better.

9.11 Pydantic as the single source of truth

Every schema decision from Part B — what a request looks like, what an error looks like, what a response envelope looks like — becomes one Pydantic class in FastAPI. The model is not documentation of the contract; it enforces it.

One model, three jobspython

from pydantic import BaseModel, Field

class CreateMessage(BaseModel):
    content: str = Field(min_length=1, max_length=10_000)
    conversation_id: str

class MessageOut(BaseModel):
    id: str
    content: str
    created_at: datetime

@app.post("/messages", response_model=MessageOut, status_code=201)
async def create_message(body: CreateMessage) -> MessageOut:
    # body is already validated: content is 1-10,000 chars, both fields present.
    saved = await db.insert_message(body)
    return MessageOut(**saved)

[def] response_model does validation on the way out, not just the way in

Without it, whatever your handler returns is serialized as-is — including any internal field you forgot to strip, like a password hash or an internal database ID scheme. With response_model declared, FastAPI filters the output to exactly that shape, silently dropping anything not declared on the model. An accidental internal field leaking to a React client is a real, common bug class that a declared response model eliminates by construction.

[retail] The same models double as your API's OpenAPI schema

CreateMessage and MessageOut are not written twice — once for validation, once for documentation. FastAPI reads the same class both times, which is the property 9.14 turns into a generated TypeScript client for the React app: the frontend's types and the backend's validation come from literally the same source, so they cannot silently drift apart the way hand-written API documentation always eventually does.

9.12 Dependency injection, done right

Nearly every endpoint needs some of the same things: the current authenticated user, a database session, a rate-limit check. FastAPI's dependency injection is the mechanism for declaring "this endpoint needs one of these" without writing that logic per handler.

A dependency, used by any endpoint that declares itpython

async def get_current_user(token: str = Depends(oauth2_scheme)) -> User:
    user = await verify_token(token)     # raises HTTPException(401) if invalid
    return user

async def get_db_session():
    async with SessionLocal() as session:
        yield session                     # code after yield run cleanup

@app.get("/conversations")
async def list_conversations(
    user: User = Depends(get_current_user),
    db: AsyncSession = Depends(get_db_session),
):
    return await db.query_conversations(user.id)

[+] A yield dependency is a context manager, spelled as a function

Code before yield runs as setup; code after it runs as teardown, guaranteed to execute even if the handler raises. This is precisely how a database session gets closed reliably without every handler remembering to close it — the dependency owns the lifecycle, the handler just uses what it's handed. It's the same acquire/release discipline chapter 7's concurrency governor needed around every in-flight request, expressed as a language feature instead of a try/finally block you have to remember to write.

[!] "function" vs. "request" scope matters the moment a response streams

A dependency's teardown normally runs after your handler function returns but before the response is sent — fine for a database session backing an ordinary JSON response. But a streaming response (9.18) starts sending bytes to the client while the handler is still a running generator; if that generator needs the database session, closing it at function-return time closes it mid-stream. Depends(get_db_session, scope="request") keeps the dependency alive until the entire response — including the stream — has actually finished sending. Missing this is a real, easy-to-hit bug the instant an endpoint needs both a database call and a streamed LLM response.

9.13 async def vs. def: the threadpool boundary

FastAPI lets you write a handler as either async def or a plain def, and treats them completely differently under the hood. Picking the wrong one for the wrong kind of work silently costs you the concurrency chapter 7 spent an entire chapter building.

[def] What actually happens to each

An async def handler runs directly on the event loop, alongside every other concurrent request, and must never block that loop with synchronous work. A plain def handler is automatically dispatched to a worker thread pool, so it can safely call blocking code without freezing the whole server — at the cost of a fixed-size pool becoming the ceiling on how many can run at once.

[!] The mistake that quietly serializes every request

Calling a blocking library — a synchronous database driver, requests.get() instead of an async HTTP client — from inside an async def handler blocks the entire event loop for every other concurrent request, not just this one. This is the async equivalent of chapter 7's warning about a default actor processing one call at a time: the whole point of going async was concurrency, and one synchronous call inside it throws that away for everyone sharing that worker process.

[+] The decision rule

I/O-bound work — a database call, an HTTP request to a vLLM fleet, a vector search — belongs in async def using an async-native library (httpx, an async DB driver). Genuinely CPU-bound work — heavy in-process computation with no async equivalent — belongs in a plain def, letting FastAPI's threadpool isolate it. Nearly everything in the scenario this chapter designs for is the first kind, which is exactly why 9.9 called async-by-default the right fit for this stack.

9.14 The OpenAPI schema as a contract

FastAPI generates a complete OpenAPI schema from your route definitions and Pydantic models automatically, with no separate documentation step. For a React frontend, that schema is more useful as a machine-readable contract than as a page of prose.

[def] Interactive docs are the visible part; the JSON schema is the useful part

Every FastAPI app serves human-readable docs at /docs for free. Less visible but more valuable for a React integration: the raw schema at /openapi.json, a complete machine-readable description of every endpoint, request shape, response shape, and error code in the API.

[+] Generating a typed client instead of hand-writing fetch calls

Tools like openapi-typescript read /openapi.json and generate TypeScript types and a typed client for the React app directly — the frontend gets compile-time errors if it sends a field the backend doesn't expect, or reads a field that doesn't exist on a response, without a human writing or maintaining those types by hand. This is the practical payoff of 9.11's point that the schema is the single source of truth: a generated client can only drift out of sync with the backend if you regenerate it against a stale schema, which is a build-process discipline problem, not a design one.

[retail] Regenerate the client in CI, not by hand, or the guarantee quietly lapses

A generated client is only as good as how recently it was regenerated. If a backend developer changes a response shape and nobody reruns the generator, the frontend keeps compiling against the old types while the actual API has moved — the exact silent-drift failure the generated client was supposed to prevent. Wiring client generation into CI, so a schema change and a stale generated client fail a build together, is what actually keeps the promise instead of just making it once at setup.

9.15 Structuring an app that will grow

A single main.py with every route in it is fine for a demo and a real liability past a dozen endpoints. FastAPI's router system is the tool for keeping the codebase organised as the API surface grows.

Routers split by resource, mounted oncepython

# routers/conversations.py
router = APIRouter(prefix="/conversations", tags=["conversations"])

@router.get("")
async def list_conversations(user: User = Depends(get_current_user)):
    ...

# main.py
app = FastAPI()
app.include_router(conversations.router)
app.include_router(messages.router)
app.include_router(generation.router)   # Part E's streaming endpoints live here

[+] A layout that scales past the demo stage

One router module per resource, shared dependencies (auth, DB sessions) defined once and imported everywhere they're needed, Pydantic models in their own module so request and response shapes aren't duplicated across routers. This is the same single-responsibility instinct as any other codebase: a file organised around one resource is easy to review as a unit, and a merge conflict in conversations.py has no way to touch generation.py.

[!] Shared dependencies belong in their own module, not copy-pasted per router

get_current_user and get_db_session should be defined exactly once, in a dependencies.py every router imports from. Redefining "how do I get the current user" slightly differently in three router files is how an auth bug ends up fixed in two of the three places it was copied — the same DRY argument this course has made about every other kind of duplicated logic, applied here to the code every single endpoint runs through.

9.16 Why request/response breaks for generation

Every endpoint in Parts B through D assumed one request produces one complete response, quickly. An LLM generation call breaks that assumption in a way that isn't a performance problem to optimise away — it's a fundamentally different response shape.

[!] A 30-second wait for one JSON blob is a bad experience even when it works

A React component awaiting a normal fetch() call shows a spinner and then the result. Applied to an LLM call generating a long answer, that means 30 seconds of a spinner and then the entire answer appearing at once — no perceived progress, no ability to start reading early, and every one of those 30 seconds is a connection that can time out at a proxy or load balancer never designed to hold a request open that long.

[def] What actually needs to change

Chapter 6 covered how vLLM generates tokens incrementally rather than waiting for the full sequence. That incremental generation is wasted on the user if the API sits in front of it and buffers the whole thing before responding. The fix is not a faster model — it's a response that can send bytes to the client as they become available, instead of only once, at the end.

[retail] This is the same tension as chapter 9's opening box, now concrete

9.1 flagged "one API surface, two response shapes" as the reason this chapter exists. Here is the shape difference exactly: a CRUD endpoint returns once; a generation endpoint needs to return repeatedly, on the same connection, until the model is done. Nothing about REST, status codes, or Pydantic models from Parts A through D breaks — they still describe the request and the eventual complete response perfectly. What's missing is a mechanism for the in-between.

9.17 Streaming choices: SSE vs. WebSockets

Two real mechanisms exist for a server to keep pushing data over one connection. Picking between them for an LLM-streaming endpoint has a clear answer once you look at what each one actually offers.

WebSockets

A full-duplex connection: both sides can send at any time, arbitrary binary or text frames, no HTTP semantics once established.

Right for: genuinely bidirectional traffic — a live collaborative document, a chat where either party interrupts mid-stream. Overkill for a client that only ever sends one request and then listens.

Server-Sent Events (SSE)

One-directional: the server streams events over a plain HTTP response the browser's native EventSource understands, with automatic reconnection built into the browser.

Right for: exactly the LLM-token-streaming shape — client asks once, server streams tokens until done. Simpler protocol, works over plain HTTP, and (9.9) is now natively supported by FastAPI.

[+] The question that settles it: does the client ever need to send more than once?

A chat completion is fundamentally one request, one streamed answer — the client doesn't need to send anything mid-generation. That is SSE's exact shape, and reaching for a WebSocket for it adds a stateful, harder-to-load-balance connection type for capability you don't use. WebSockets earn their complexity when the client genuinely needs to interrupt, redirect, or send follow-up input while the server is still mid-response — a voice interface, a collaborative canvas — which most chat-style LLM UIs simply don't need.

[!] Both share the same infrastructure trap

A corporate proxy, an older load balancer, or an aggressive reverse-proxy timeout can all silently kill a long-lived connection of either kind, and the failure looks identical from the React app: the stream just stops. This is why 9.18's implementation includes a keep-alive ping — not a nicety, but the concrete fix for a connection an intermediary decided was idle and closed.

9.18 Server-sent events, concretely

FastAPI 0.141's own fastapi.sse module (9.9) turns this from a protocol you'd otherwise hand-roll into a few lines against a documented API.

Streaming tokens from a vLLM call as they arrivepython

from fastapi.sse import EventSourceResponse, ServerSentEvent

@router.post("/chat/completions", response_class=EventSourceResponse)
async def stream_completion(body: ChatRequest, gateway=Depends(get_vllm_gateway)):
    async def event_stream():
        async for token in gateway.stream_generate(body.messages):
            yield ServerSentEvent(data={"token": token})
        yield ServerSentEvent(event="done", data={"finish_reason": "stop"})

    return EventSourceResponse(event_stream())

[def] What the browser actually receives

Each ServerSentEvent is encoded to the SSE wire format: a data: line (JSON-encoded), an optional event: line naming the event type, and a blank line terminating it. React's EventSource API — or a small wrapper around fetch with a streaming body, more common when auth headers are needed since EventSource can't set them — parses this natively, no custom protocol required on the frontend.

[+] A named "done" event is what tells React the stream actually finished

Without an explicit terminal event, a React client can't distinguish "the model finished normally" from "the connection dropped after the last token happened to arrive" — both look identical from the client's side otherwise. An explicit event: done carrying a finish_reason (stop, length, content_filter) gives the frontend an unambiguous signal to stop showing a loading state, and a reason to show if it wasn't a clean stop.

[retail] Keep-alive pings prevent the exact proxy timeout 9.17 warned about

If token generation stalls for a few seconds — a slow tool call mid-agent-loop from chapter 8, a queued request behind chapter 7's concurrency governor — a proxy with no traffic on the connection may decide it's dead and close it. A periodic SSE comment line (: ping, ignored by EventSource but enough to count as traffic) sent every 10–15 seconds keeps the connection looking alive to anything watching for silence, without affecting what the client actually renders.

9.19 Long-running inference and backpressure

Streaming solves the perceived-latency problem. It does not solve what happens when a generation call genuinely takes minutes — a long agentic task from chapter 8, not a single chat turn — or what happens when the inference engine itself is the bottleneck, not the network.

[def] Three shapes for "this will take a while"

Stream and hold the connection: 9.18's approach. Simplest, but the client must stay connected the entire time — a closed laptop lid or a mobile app backgrounded mid-stream loses the connection and, typically, the result.
Background task, poll for status: The endpoint returns immediately with a job ID via FastAPI's BackgroundTasks — fire-and-forget work scheduled to run after the response is sent — or a real queue for anything that must survive a process restart; the client polls GET /jobs/{id} until it's done. Survives a disconnected client, at the cost of polling overhead and a job store to track state.
Background task, notify via webhook or a second SSE stream: The client subscribes once and gets pushed a completion event rather than polling — the better trade for a job that reliably takes minutes rather than seconds, where polling would mean many wasted requests.

[!] The bottleneck may not be your API at all

If chapter 7's concurrency governor is refusing new requests because the vLLM fleet is already at its ceiling, that's not your API failing — it's correctly reporting upstream saturation. Returning a generic 500 here is actively misleading; a 503 with the model_unavailable code from 9.6, ideally with a Retry-After header if you can estimate one, tells the React client the truth: this isn't broken, it's full, try again shortly.

[+] Give the client a way to cancel, not just a way to wait

A user who closes a chat mid-generation and starts a new question is a case worth designing for explicitly: if the client disconnects an SSE stream, the handler should detect it (FastAPI exposes request.is_disconnected() for exactly this) and stop consuming GPU capacity generating tokens nobody will read. Without it, an abandoned request keeps a slot occupied against chapter 7's max_in_flight ceiling for no benefit to anyone.

9.20 CORS: letting the React app in

A React dev server on localhost:5173 calling an API on localhost:8000 is a cross-origin request by the browser's definition, and the browser blocks it by default. CORS is the mechanism for telling the browser which origins are allowed to call this API at all.

[def] The same-origin policy is the browser's default, not the server's

Two URLs share an origin only if the scheme, host, and port all match exactly. localhost:5173 and localhost:8000 are different origins despite sharing a hostname. This restriction exists in the browser to stop a malicious page from silently reading responses from a bank's API in another tab; CORS headers are the server explicitly opting specific origins back in.

Allowing the React app's dev and production originspython

from starlette.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:5173", "https://app.example.com"],
    allow_credentials=True,          # needed if the client sends cookies
    allow_methods=["GET", "POST", "PUT", "DELETE"],
    allow_headers=["Authorization", "Content-Type"],
)

[!] allow_origins=["*"] and allow_credentials=True cannot both be true

The wildcard origin combined with credentialed requests would let any site on the internet make authenticated calls against your API using a logged-in user's cookies — the browser spec forbids the combination outright, and most CORS implementations enforce it. List actual origins explicitly once credentials are involved. A wildcard is only safe for a fully public, unauthenticated API.

[retail] A CORS error is almost never actually about CORS

The browser console error is unhelpful by design — it won't tell you which origin, method, or header was rejected, only that something was. In practice the fix is nearly always one of: the deployed frontend's actual origin isn't in allow_origins, a header the client sends (Authorization, a custom header) isn't in allow_headers, or the request is a "preflighted" one (a non-simple method or header) and the OPTIONS preflight itself is failing before the real request is even sent.

9.21 Authentication and authorization

Authentication answers "who is this." Authorization answers "what are they allowed to do." Conflating them is how a 401 gets returned for a permissions problem, which 9.2 already flagged as actively misleading to a client trying to react correctly.

The three shapes an API token actually takes
Scheme	Good fit for	The trade-off
Session cookie	A traditional web app where the browser and API share a domain	Automatic on every request, but needs CSRF protection and doesn't fit a mobile client cleanly.
JWT bearer token	A React SPA calling a separately hosted API — this chapter's scenario	Stateless and easy to verify without a database round-trip, but hard to revoke before it expires — short expiry plus a refresh token is the usual fix.
API key	Server-to-server or third-party integration access, not an end user's browser session	Simple, but a static long-lived secret; leaking one is a bigger blast radius than a short-lived token leaking.

Authorization as a second, separate dependencypython

async def require_owner(
    conversation_id: str,
    user: User = Depends(get_current_user),   # authentication (9.12)
    db: AsyncSession = Depends(get_db_session),
) -> Conversation:
    conv = await db.get_conversation(conversation_id)
    if conv is None:
        raise HTTPException(404, detail="not found")
    if conv.owner_id != user.id:
        raise HTTPException(403, detail="not your conversation")   # authorization
    return conv

[+] Return 404, not 403, when even confirming existence is a leak

For a resource whose existence itself is sensitive — another user's private conversation — a 403 confirms to an attacker that the ID they guessed is real, just not theirs. Returning 404 for "exists but not yours" as well as "doesn't exist at all" leaks nothing extra. This is a judgment call, not a rule: a 403 is more honest and more debuggable when the resource being forbidden isn't itself sensitive information.

9.22 Rate limiting and idempotency

Two different problems get solved by two related mechanisms: stopping a client from sending too much, and making sure a client's retry after a dropped connection doesn't double an action that already happened.

[def] Rate limiting protects the server; idempotency protects the client's retry

A rate limit exists so one client (or one attacker) can't monopolise capacity that others need — chapter 7's max_in_flight ceiling protects the vLLM fleet the same way, one layer down. Idempotency exists because networks are unreliable: a client that sends a request and never sees the response genuinely cannot tell whether it succeeded, and the only safe action is to retry — which is only safe for the server if the retry is recognised as the same request, not a new one.

An idempotency key, enforced server-sidepython

@router.post("/payments")
async def create_payment(
    body: CreatePayment,
    idempotency_key: str = Header(...),
    db: AsyncSession = Depends(get_db_session),
):
    existing = await db.get_by_idempotency_key(idempotency_key)
    if existing is not None:
        return existing              # same request seen before - return the original result
    result = await process_payment(body)
    await db.save_idempotency_key(idempotency_key, result)
    return result

[retail] This is chapter 8's idempotency key, on the other side of the same call

Section 8.5 covered giving an agent's side-effecting tool call a stable idempotency key so a retried tool call couldn't double a refund. This is the exact same mechanism from the API's side of that call: whether the retry originates from a flaky network, a React client's own retry logic, or an agent retrying a tool call, the API has to honour the same key the same way regardless of who's calling. Idempotency has to be designed into whichever layer actually performs the side effect, and it only works if every caller of that layer is disciplined about sending the key.

[!] A 429 without Retry-After makes the client guess

Rate-limiting a client without telling it when to try again invites either an immediate retry storm (making the overload worse) or an overly conservative backoff (wasting capacity that's actually free again). A Retry-After header with a concrete number of seconds turns a vague "slow down" into an actionable instruction the client's retry logic can follow exactly.

9.23 Observability and testing before shipping

An API that works in a demo and an API you can debug at 2am under real load are built differently. The gap between them is almost entirely structured logging, request tracing, and tests that run against the actual contract rather than the implementation.

[def] A request ID is the thread that ties one user's problem report to your logs

Every request should get a unique ID at the moment it arrives, logged with every subsequent line touching that request, and returned to the client in a response header. When a user reports "my message didn't send," that ID is what lets you find the exact request in logs spanning your API, the database call, and the vLLM gateway from chapter 7 — without it, you're grepping timestamps and guessing.

[+] Log the shape of the request, never its content, for an LLM endpoint

Logging "generation request: 340 tokens in, 512 tokens out, 2.1s, gateway saturated: false" is useful for debugging and safe to retain. Logging the actual prompt and completion text by default is not — it may contain whatever a user typed, including things they never intended to have stored durably in a log aggregator with a different retention policy and access list than your primary database. Log structure and metrics by default; log content only behind an explicit, access-controlled debug flag.

[retail] Test the contract with FastAPI's TestClient, not just the happy path

TestClient lets you call an endpoint in-process, no server needed, and assert on the exact response shape a React client will actually receive — status code, error code field from 9.6, response schema from 9.11. The tests worth writing are the boundary cases: a missing required field returns 422 with the right error shape, an unauthenticated request returns 401 not a stack trace, a request for someone else's resource returns 403 or 404 per 9.21's decision, not 200 with the wrong data. Each of those is a one-line regression against a real security or correctness property, not busywork.

9.24 Key takeaways

The twelve things worth remembering

The API is a translation layer, not "the backend." Every system it fronts — database, vector index, vLLM fleet, agent loop — already exists from earlier chapters; the API's job is speaking HTTP on one side and each system's native protocol on the other.
HTTP methods are promises, not suggestions. GET is safe, PUT and DELETE are idempotent by convention, and every client, proxy, and retry mechanism between the browser and your server relies on those promises holding.
URLs name resources; methods supply the verb. A frontend that can guess an endpoint's shape from the noun and the HTTP verb conventions doesn't need to read documentation for the obvious cases.
Validate at the edge, trust everything after it. A schema declared once at the boundary is what lets every function behind it assume well-formed input instead of re-checking types three layers deep.
An error response is part of the contract. A stable machine-readable code plus a human-readable message lets a frontend branch on behaviour without parsing prose, and an LLM-call failure needs distinct codes because the right client reaction differs by cause.
Design pagination, filtering, and versioning in from day one. A cursor survives inserts an offset can't; a URL version prefix costs nothing before you need it and saves a painful retrofit after.
FastAPI's whole value proposition is one type hint doing three jobs. The same annotation drives validation, OpenAPI schema generation, and response serialization — nothing is declared twice.
Dependency injection is how cross-cutting logic stops being copy-pasted. Auth, database sessions, and rate limits are declared once and requested by any endpoint that needs them; a yield dependency's scope must be widened to "request" the moment a response streams.
async def and def are not interchangeable performance options. Blocking code inside async def stalls every concurrent request sharing that worker, not just its own.
A generation endpoint is not a slow JSON endpoint — it's a different response shape. Streaming via Server-Sent Events, not a longer wait, is the actual fix, and FastAPI 0.141 ships SSE support natively.
Idempotency keys protect a retry on both sides of the same call. Chapter 8's agent tool retries and a React client's network retry both need the same mechanism: the API must recognise a repeated key as the same request, not a new one.
CORS, rate limiting, and structure are what separate a demo from a production API. None of them are optional additions for "later" — a demo without them just hasn't hit the failure mode yet that makes them necessary.

[def] The one-sentence version

Design the contract — resources, validation, errors, pagination, versioning — before picking a framework; let FastAPI's type-hint-driven validation and dependency injection implement that contract with almost no duplicated logic; and treat any endpoint that calls an LLM as a genuinely different response shape, not a slow version of the same one.

9.25 Interview drills

API design questions test whether you've actually built one someone else depended on, or only ever consumed one. Every answer below starts from a concrete failure mode, not a definition.

1. Why does it matter that PUT is idempotent and POST isn't?

Because every layer between the browser and my server — the client's own retry logic, a corporate proxy, a load balancer — is allowed to assume that promise and act on it. If a PUT times out after the server actually processed it, a client retrying it is safe, because replaying the same replacement twice leaves the same end state as once. Retrying a POST that already succeeded can create a duplicate resource, because nothing about POST promised otherwise.

The mistake I'd watch for is writing a handler that violates the method's promise — a PUT that appends instead of replaces. The moment that happens, every automatic retry mechanism relying on the HTTP spec's promise becomes a data corruption risk instead of a safety net.

2. A client gets a 401 when they're logged in correctly but lack permission for the resource. What's wrong, and why does it matter?

401 means "I don't recognise your credentials at all"; 403 means "I know exactly who you are, and the answer is no." Returning 401 for a permissions failure is a semantic bug: it tells the client, and potentially an attacker, that the fix is to re-authenticate, when re-authenticating changes nothing because the account genuinely lacks the permission.

Practically, a React app that treats 401 as "redirect to login" will now redirect an already-logged-in user to a login screen for a request that was never going to succeed no matter how many times they log in, which is a confusing and wrong user experience traceable directly to using the wrong status code.

3. Where should request validation live, and why not just check things inside the handler function?

At the boundary, as a schema, before the handler's actual logic runs. The value isn't just avoiding repetition — it's that everything behind the validation step can then assume well-formed input as an invariant rather than a hope. If three different handlers each check "is this actually a string" by hand, that's three places to get subtly wrong, and three places that drift out of sync the first time one of them is updated and the others aren't.

I'd also distinguish this from validating meaning, not just shape: a schema confirms a field is a string under 10,000 characters, not that it's a sensible thing to send an LLM. Both matter, but they're different checks at different layers, and conflating them is how "it passed validation" gets mistaken for "it's safe to act on."

4. Why wrap a list endpoint's response in an envelope instead of returning a bare JSON array?

Because a bare array has nowhere to put anything that isn't one of the items — a total count, a next-page cursor, a flag saying more data exists. The day pagination becomes necessary, a bare-array response has no way to add it without changing the response's fundamental type, which every existing client parsing it as an array will break on.

Wrapping it in {"data": [...], "meta": {...}} from the start costs nothing when meta is empty, and means adding pagination later is an additive change to a field nobody was reading yet, not a breaking change to the response's shape.

5. What does one type hint actually do in a FastAPI path operation, mechanically?

It's read at import time and used for three separate things from the same annotation: building a Pydantic validator that runs before the handler executes, contributing to the OpenAPI schema generated for that route, and — on a return annotation or response_model — filtering and serializing whatever the handler returns down to exactly that shape.

The practical implication I'd highlight: because validation, docs, and serialization all come from the same class, they cannot silently drift apart from each other the way a hand-maintained documentation page can drift from the actual code. That's also what makes generating a typed TypeScript client from the schema trustworthy rather than a snapshot that goes stale.

6. An endpoint needs both a database session and a streamed SSE response. What goes wrong if you don't think about dependency scope?

By default, a yield dependency's teardown runs after the handler function returns but before the response is actually sent to the client. For an ordinary JSON response that's invisible and correct. For a streaming response, the handler function is a generator that's still running — still yielding chunks — long after it "returns" in that sense, so a database session torn down at that point gets closed while the stream is still trying to use it mid-generation.

The fix is declaring that dependency with request scope instead of the default function scope, so its teardown waits for the entire response, streaming included, to actually finish. I'd flag this as exactly the kind of bug that only shows up once streaming and a resource dependency combine on the same endpoint — each works fine in isolation.

7. When would you choose def over async def for a FastAPI handler?

When the handler does genuinely CPU-bound work with no async-native equivalent — heavy in-process computation, not an I/O wait. A plain def handler gets automatically dispatched to a worker thread pool, so it can run blocking code without freezing the event loop that every other concurrent async request is sharing.

The mistake I'd specifically watch for is the reverse: calling a blocking library — a synchronous DB driver, a synchronous HTTP client — from inside an async def handler. That blocks the whole event loop for every other request during the call, silently serialising a server that was supposed to handle hundreds of requests concurrently. Nearly everything in a typical API — database calls, HTTP calls to an inference fleet — is I/O-bound and belongs in async def with an async-native library instead.

8. Your product wants to stream an LLM's response to a chat UI. Why SSE over WebSockets?

Because the actual traffic pattern is one-directional: the client sends one request and then only listens until the answer finishes. That's exactly what Server-Sent Events model — a plain HTTP response the browser's EventSource API understands natively, with automatic reconnection built in, no custom protocol needed on either end.

A WebSocket buys full duplex communication I don't need for this and costs a more stateful connection that's genuinely harder to load-balance and scale. I'd reach for a WebSocket specifically when the client needs to send something mid-response — interrupt generation, redirect it, voice input arriving while the model is still talking — which most chat-style LLM UIs simply don't do.

9. Your vLLM fleet from chapter 7 is at its concurrency ceiling and rejecting new requests. What should your API return, and why not a generic 500?

A 503, not a 500, because my API isn't broken — it's correctly reporting that something it depends on is saturated. A 500 tells whoever's debugging this to look at my code first, which wastes their time; a 503 correctly points them upstream. I'd also give it a specific error code like model_unavailable rather than a generic failure, and a Retry-After header if I can estimate one, so the client's retry behaviour can be informed instead of guessed.

I'd also want the endpoint to detect a client that's disconnected mid-stream and stop generating for them, because an abandoned request still holding a slot against that same concurrency ceiling is actively making the saturation problem worse for everyone else waiting behind it.

10. A React client's request appears to have failed, but it can't tell whether the server actually processed it before the connection dropped. How do you make retrying safe?

With a client-generated idempotency key sent on the request, checked server-side before doing the work: if a request with that key has already been processed, return the stored result instead of processing it again. That converts "did this actually happen" from a genuine unknown into a safe question to re-ask, because the server recognises the retry as the same logical request rather than a new one.

I'd connect this to chapter 8's agent tool calls directly: an agent retrying a side-effecting tool call after a timeout needs exactly the same mechanism, and in both cases the discipline has to live in whichever layer actually performs the side effect — a payment, a refund, a message send — not in the caller, because the caller is precisely the thing that can't be trusted to know whether its previous attempt succeeded.

Where this leaves you

You can now design the layer every other chapter's system actually sits behind: a resource-shaped contract with validation and errors a frontend can build against, a reasoned choice of framework rather than a default one, and the specific mechanics — dependency scope, the async/def boundary, server-sent events — that make the difference between an API that happens to work in a demo and one that holds up under a real React client with real users behind it.

The thread running through this chapter was the same one from 9.1: one API surface has to serve two genuinely different response shapes, and pretending an LLM call is just a slow JSON endpoint is where most of the actual bugs in this kind of system come from. Everything from streaming to backpressure to cancellation in Part E exists because that distinction was taken seriously instead of engineered around.

That closes out the AI stack proper: from what a language model is and how a transformer works, through embeddings, vector search, and retrieval-augmented generation, into serving those models efficiently on real hardware, orchestrating a fleet of them elastically, letting them act on the world with appropriate care, and finally the API layer a real frontend actually calls. The next chapter steps to the other side of that connection: React, the framework a real user's browser runs, set against the wider frontend framework landscape, and the discipline a streaming AI chat interface needs on top of the SSE endpoint built here.