Chapter 16 — Microservices and System Design

[!] The bias this chapter is written against

Most microservices material is written by people selling microservices. It presents the split as inevitable and treats the monolith as a phase you grow out of. That is not what the evidence says, and it is not what senior interviewers reward. The strong answer is almost always a smaller number of services than the candidate proposes, with clear reasons for each cut.

So this chapter spends its first section arguing against splitting, and returns to that scepticism throughout. Distribution is a cost you pay to buy something specific. If you can't name what you're buying, don't pay.

[i] What this assumes, and what it doesn't repeat

It assumes the components built earlier: the vector database from chapter 4, the RAG pipeline from 5, the vLLM server from 6, the agent loop from 8, the FastAPI service from 9, the storage decisions from 11, and the deployment substrate from 13. Those are the boxes; this chapter draws the arrows.

Deliberately not repeated here: API contract design (9.3 to 9.7), authentication (9.21), Kubernetes mechanics (14.6 to 14.12), and the guardrail checkpoints (15.11 to 15.17). They are referenced where a design depends on them, rather than re-explained.

16.1 The monolith you should probably keep

A microservice architecture buys you independent deployment, independent scaling, and independent failure. It costs you a network between every component. Whether that trade is worth it depends entirely on whether you need what it buys.

[def] The distributed computing tax

Turning a function call into a network call changes its properties in five ways at once. It can now be slow (milliseconds instead of nanoseconds), fail independently of the caller, partially succeed, arrive out of order, and arrive more than once. None of these were possible in-process.

Every one of those needs code: timeouts, retries, idempotency keys, circuit breakers, tracing. That code is the tax. Sections 16.8 to 16.12 are almost entirely about paying it.

[hist] The pendulum, briefly

Microservices became mainstream around 2014, largely through Netflix and Amazon engineering writing. By roughly 2020 the industry had accumulated enough failed splits to produce a counter-movement, including some well-publicised consolidations back to fewer services. The lasting lesson was not "microservices are bad" but that the organisations that succeeded had a specific scaling or team-autonomy problem, and the ones that struggled were copying an architecture without inheriting the problem it solved.

What actually justifies a split

Reasons to extract a service, ranked by how well they hold up
Reason	Verdict
It needs different hardware. A GPU inference server and a CRUD API cannot share a process, and you do not want to buy GPUs to scale JSON handling.	Strongest reason. Physical, unarguable, and the main one in AI systems.
It scales on a different axis. Embedding throughput follows catalogue size; chat concurrency follows active users. Those numbers move independently.	Strong — if the numbers genuinely diverge. Check rather than assume.
A separate team owns it, with its own release cadence and on-call rota.	Strong. This is Conway's law used deliberately rather than suffered.
It has a different availability requirement. Checkout must survive recommendations being down.	Strong, and it forces you to design the degraded path explicitly.
It has a different release risk. Prompt changes ship several times a day; the payment path ships rarely and carefully.	Reasonable, though feature flags often solve this without a network hop.
The codebase is large.	Weak. Large codebases need modules, not networks. A distributed monolith is strictly worse than a big one.
Microservices are the modern approach.	Not a reason. This is the one that produces the systems people later write conference talks about consolidating.

[!] The distributed monolith

The common failure is not "too many services." It is services that were split physically but not logically: they deploy separately, but every change touches three of them and they must be released in a specific order.

You have built one if any of these are true: a schema change requires a coordinated multi-service release; services share a database; you cannot deploy service A without first deploying B; or a single user action fans out into a synchronous chain of five internal calls. This has every cost of distribution and none of the independence.

[+] The modular monolith, which is the right default

Enforce module boundaries inside one deployable: separate packages, explicit interfaces between them, no reaching into another module's tables. You get the boundary discipline, and calls stay in-process — fast, transactional, and debuggable with a stack trace.

The payoff is that these boundaries are the cheapest possible place to be wrong. Move a boundary inside a monolith and it is a refactor your compiler helps with; move one between services and it is a migration with dual writes and a backfill. Get the boundaries right in-process, then extract the pieces that have earned it from the table above.

[retail] What the running example justifies

The retail platform across chapters 4, 5, 8 and 9 splits into roughly five services, and it is worth being precise about which reason each one invokes:

LLM inference (chapter 6) — different hardware. GPUs.
Embedding (chapter 3) — different hardware and a different scaling axis: batch catalogue re-embedding versus per-query latency.
Vector search (chapter 4) — different hardware again, but memory rather than GPU: a 269 GB working set that must stay resident.
The API and orchestration layer (chapter 9) — ordinary CPU, scales with user concurrency.
Catalogue and orders (chapter 12) — transactional Postgres work with a stricter availability requirement than anything AI-related.

[!] Notice what is not on that list

Reranking, prompt construction, guardrail checks and conversation memory are all libraries inside the orchestration service, not services of their own. They are pure CPU, they scale with the same traffic, and no separate team owns them. The source syllabus lists them as candidate services, and a weak interview answer draws each as its own box. Making the prompt service a network hop adds a failure mode and a few milliseconds to buy precisely nothing. Section 16.13 revisits this properly.

16.2 What a service boundary actually is

Given that you are splitting something, the question is where to cut. The wrong cut is worse than no cut, because it is expensive to undo.

[def] The test: what changes together, stays together

A boundary is well-placed when a typical business change lands inside exactly one service. If adding a field to a product means editing the catalogue service, the search service and the recommendation service, that boundary is wrong regardless of how clean the diagram looks. High cohesion inside, low coupling across — the same principle as module design, with a much higher penalty for getting it wrong.

[!] The three cuts that look right and aren't

By technical layer. A "database service", an "API service", a "business logic service". Every feature now touches all three. This is a distributed monolith with extra steps.
By entity. One service per database table — a user service, an address service, a preference service. These have no independent reason to exist and chat constantly. The give-away is a service whose API is just CRUD on one table.
By what the org chart looked like eighteen months ago. Boundaries outlive the teams that motivated them. Revisit them when the teams change, or you get services nobody owns.

[+] Cut by capability instead

A capability is something the business would recognise as a thing it does: "search the catalogue", "take payment", "answer a support question". These are stable — the business still does them after a reorganisation — and they own their data end to end. In domain-driven design terms this is a bounded context, and the useful part of that idea is the observation that the same word means different things in different contexts, and that is fine.

[retail] "Product" means four different things

In the retail platform, a product is:

To the catalogue service: a row with a title, price, stock level, supplier and tax category. Roughly forty fields.
To the search service: an ID, a 256-dimension vector, a market and a handful of filterable attributes (4.12).
To the support agent: a name and a return policy.
To the order service: an ID, a price captured at purchase time, and a quantity.

The instinct is to build one canonical Product model shared by all four. Resist it: that shared model becomes a coupling point that every service must agree on before it can change, which is exactly the independence you split to get. Let each context keep the narrow version it needs, and accept the duplication — it is cheaper than the coordination.

16.3 Database per service

This is the rule people most often skip, and skipping it is what turns a service split into theatre.

[def] The rule, and the reason behind it

A service owns its data exclusively. No other service reads its tables directly; they go through its API. The reason is not purity — it is that a shared database is a shared schema, and a shared schema cannot be changed independently. If another team's report breaks when you rename a column, you do not have an independent service, whatever the deployment diagram says.

[!] What you give up, stated honestly

Two things, and both are genuinely painful:

Joins across services. "Orders with product titles for market MX" was one SQL query. Now it is a query plus N API calls, or a denormalised copy. There is no third option that preserves both independence and the join.
Transactions across services. You cannot roll back another service's commit. This is what section 16.11 is about, and it is the single biggest cost of the split.

[+] Separate schemas as a staging post

You do not need separate database servers to get the discipline. Separate schemas in one Postgres instance, with per-service credentials that only grant access to their own schema, enforce the boundary while keeping one thing to operate and back up. The permission grant is what makes it real: without it the rule is a convention, and conventions lose to deadlines. Split the instance later if a service genuinely needs different hardware or availability.

[retail] The vector index is a derived copy, not a source of truth

The search service's Qdrant collection holds product IDs, vectors and filter attributes — all of it derived from the catalogue service's Postgres. This is the "system of record versus derived copy" distinction from 12.4, and it matters here because it tells you what to do when they disagree: rebuild the copy, never reconcile. The catalogue is right by definition. The alias-swap re-index from 4.16 is the recovery procedure, and knowing which store is authoritative is what makes that a safe decision rather than a frightening one.

16.4 Discovery and the API gateway

Once there is more than one service, two questions appear: how do services find each other, and how does the outside world reach any of them.

[def] Service discovery

Instances come and go — autoscaling, rescheduling, crashes — so their addresses cannot be configuration. Discovery is the registry that maps a stable name to the current set of healthy addresses. On Kubernetes this is built in: a Service object gives you a DNS name that resolves to healthy pods, and the readiness probes from 14.10 are what decide "healthy". If you are on Kubernetes, you already have service discovery and should not add another system for it.

[def] API gateway

One public entry point in front of many internal services. It handles the concerns you do not want implemented seven times: TLS termination, authentication, rate limiting, routing, and request logging.

The value is consolidation of cross-cutting concerns, not routing. Routing is the easy part.

What belongs at the gateway, and what does not
At the gateway	In the service
TLS termination	Business logic of every kind
Authentication: is this token valid? (9.21)	Authorization: may this user touch this order? The gateway does not know.
Coarse rate limiting and quota (9.22)	Fine-grained limits that depend on domain state, such as per-tenant token budgets
Routing, and request/trace ID injection (16.12)	Response shaping and pagination

[!] Two ways gateways go wrong

It accumulates logic. A transformation here, a special case there, and eventually deploying anything requires a gateway change. It has become a monolith that every team must queue for — and it is on the critical path of every single request.
It becomes a single point of failure. Everything goes through it, so it must be more available than anything behind it: multiple replicas, spread across zones, with a deployment process more conservative than the services it fronts.

[+] Do not confuse the API gateway with the model gateway

They sit at different edges and solve different problems. The API gateway faces your users and guards the way in. The model gateway (16.14) faces your LLM providers and guards the way out — keys, spend, failover between vendors. An interview answer that merges them into one box will be asked about token budgets and provider failover, and the merged box has no good answer.

16.5 Synchronous calls and their failure modes

A synchronous call is the one everyone reaches for first, because it looks like the function call it replaced. That resemblance is the problem.

[def] Availability multiplies down a synchronous chain

If A calls B synchronously and waits, then A cannot be more available than B. Chain them and the arithmetic is unkind: five services at 99.9% each give 0.999⁵, which is 99.5% — roughly 3.6 hours of downtime a month rather than 43 minutes.

Worse, latency adds while availability multiplies, and the tail is what users feel. If each hop has a p99 of 100 ms, the chain's p99 is far worse than 500 ms, because you are now exposed to whichever hop is having its bad moment.

[!] The fan-out chain is the anti-pattern to name in an interview

One user request triggering a synchronous chain of five internal calls is the design most likely to be criticised, and correctly. It couples availability, adds latency at every hop, and makes debugging require five sets of logs. The fix is usually to ask which of those calls the user is actually waiting for. Often only the first two matter, and the rest can be events (16.6).

REST, gRPC, or GraphQL between services

Choosing a synchronous protocol for internal traffic
Protocol	Use it when	Cost
REST/JSON	The default. Debuggable with curl, readable in logs, universally supported.	Verbose on the wire; no enforced schema unless you add one (9.14).
gRPC	High-volume internal calls where latency matters. Binary, HTTP/2 multiplexed, schema-enforced by protobuf, with streaming built in.	Not human-readable; needs codegen; awkward through browsers and some proxies.
GraphQL	At the edge, when varied clients need different field subsets from many sources.	Between internal services it is usually a mistake — it moves query complexity to the caller and makes rate limiting and caching much harder.

[retail] Where gRPC earns its keep here

The orchestration service calls the embedding service on every single search query, with a fixed request shape and a large float array coming back. That is precisely gRPC's case: the payload is binary-friendly, the schema never varies per call, and saving 3 to 5 ms per hop is material when the whole search budget is 200 ms. The public search endpoint stays REST, because it faces the React app in chapter 11 and the browser is not where you want protobuf.

16.6 Asynchronous and event-driven

The alternative to waiting is not waiting. Instead of calling a service, you publish a fact and move on.

[def] Commands versus events

A command tells a specific service to do something: ReindexProduct(id). It has one intended recipient and implies the sender knows who should act.

An event states that something happened, in the past tense: ProductUpdated(id, fields). The publisher does not know or care who consumes it. This is the distinction that determines coupling — events let you add a consumer without touching the producer, commands do not.

[+] What asynchrony actually buys

The consumer can be down. Messages queue up. The producer neither knows nor cares, and nothing fails while the consumer is redeployed.
Load levelling. A burst of 50,000 catalogue updates does not become 50,000 simultaneous embedding requests; the queue absorbs the spike and consumers drain it at their own rate. For GPU work this is the difference between a queue and an outage.
Adding consumers is free. A new analytics service subscribes to OrderPlaced with no change to the order service — and, importantly, no new way for the order service to fail.

[!] What it costs, which is mostly certainty

Eventual consistency. A product updated at 12:00:00 may be searchable at 12:00:04. You must decide what the user sees in between, and "it will be fine" is not a decision.
Debugging gets harder. There is no stack trace across a queue. Without correlation IDs (16.12) you are reconstructing causality from timestamps.
Failure moves. A synchronous call fails in front of the user, who retries. An async failure happens later, invisibly, and needs a dead-letter queue and someone watching it.
Ordering is not free. Two updates to the same product can be processed out of order and leave the wrong value. Section 16.8 deals with this.

[retail] Choosing per-path, not per-system

The same platform uses both, and the deciding question is always "is the user waiting?"

Search query to vector search: synchronous. The user is staring at a spinner. An event here is absurd.
Catalogue update to re-embedding: asynchronous. Nobody is waiting, it is GPU work with a spiky arrival pattern, and a few seconds of staleness is invisible.
Order placed to confirmation email: asynchronous. The email service being down must never fail a purchase.
Order placed to payment authorisation: synchronous. The user must be told now whether their card worked.

16.7 Kafka and RabbitMQ, chosen properly

These get compared as if they were competitors. They are different data structures with different purposes, and the choice is usually obvious once you say which one you need.

[def] A log versus a queue

Kafka is a distributed append-only log. Messages are written to a partition and stay there for a configured retention period. Consumers track their own offset, so reading does not remove anything — a second consumer group reads the same messages independently, and either can rewind and replay history.

RabbitMQ is a message broker. A message is routed to a queue, delivered to one consumer, acknowledged, and then gone. It is built for per-message delivery and routing, not for retaining history.

Kafka 4.3 and RabbitMQ 4.3 on the axes that decide it
	Kafka	RabbitMQ
Model	Partitioned log; consumers own offsets	Queues with exchange-based routing
Replay	Yes — reset the offset and reprocess	No. Once acknowledged, it is gone
Ordering	Guaranteed within a partition	Per queue, easily lost with competing consumers
Routing	Basic: topic and partition	Rich: direct, topic, fanout, headers, priorities
Scale	Very high throughput; scales by adding partitions	High, but not log-scale
Operational weight	Heavier, though much lighter since 4.0 removed ZooKeeper	Lighter to run and reason about

[hist] Two changes that date older advice

Kafka 4.0 removed ZooKeeper entirely. KRaft mode, where Kafka manages its own metadata quorum, is now the only option — the release simply no longer ships a ZooKeeper configuration. Any tutorial that starts by launching ZooKeeper predates this, which is a useful staleness signal when you are reading advice online.

RabbitMQ 4.0 removed classic queue mirroring after years of deprecation. Replicated messaging now means quorum queues (Raft-based) or streams. Streams are worth knowing about in this comparison: they are RabbitMQ's append-only, replayable log type, which narrows the gap with Kafka for consumers that need to re-read.

[+] The short decision rule

Choose Kafka if multiple independent consumers need the same stream, if you need to replay history to rebuild a derived store, or if ordering per key matters. Choose RabbitMQ if you are distributing discrete tasks to workers, need complex routing or priorities, and each message has exactly one rightful owner. If the honest answer is "a few thousand messages a day and one consumer", choose neither — a Postgres table polled as a queue is a legitimate architecture that many teams outgrow much later than they expect.

[retail] Kafka for the catalogue stream, and why replay is the reason

product.updated goes to Kafka, partitioned by product ID so that all updates to one product land in one partition and stay ordered. Three consumer groups read it independently: the embedding pipeline, the search indexer, and analytics. The decisive factor is replay: when the embedding model is upgraded (3.x) the entire catalogue must be re-embedded, and with Kafka that is resetting an offset rather than building a bespoke backfill job. RabbitMQ would have made that a project.

16.8 Delivery semantics and idempotency

Every message system advertises a delivery guarantee. Understanding which one you actually have determines how much defensive code you need to write.

[def] The three guarantees

At-most-once: Fire and forget. Messages can be lost, never duplicated. Acceptable for metrics and low-value telemetry; unacceptable for anything a user would notice.
At-least-once: Retried until acknowledged. Nothing is lost, but duplicates happen. This is what you have in practice, and what you should design for by default.
Exactly-once: Each message takes effect once. Real within a single system's boundary — Kafka offers it for read-process-write cycles inside Kafka — but the moment your handler charges a card or calls an external API, that guarantee does not extend to the side effect.

[!] Why exactly-once cannot be bought end to end

Consider a consumer that processes a message, performs a side effect, then acknowledges. If it crashes between the side effect and the acknowledgement, the message is redelivered and the side effect happens twice. Move the acknowledgement first and a crash loses the work instead. There is no ordering of those two steps that is safe, because they are in different systems and cannot share a transaction. The problem is not solvable by configuration; it is solved by making the side effect idempotent.

[+] Idempotency, and the four ways to get it

An operation is idempotent when performing it twice leaves the same state as performing it once. Ranked by how much work they take:

Be naturally idempotent. SET stock = 12 is safe to repeat; stock = stock - 1 is not. Prefer absolute values to deltas wherever the domain allows it — this is free and most teams miss it.
Upsert on a natural key. Writing a vector for product 4471 to Qdrant is an upsert by point ID; doing it twice is harmless. Much of the retail pipeline is safe for exactly this reason.
Deduplicate on a message ID. Record processed IDs in a table with a unique constraint, inside the same transaction as the work. A duplicate violates the constraint and is skipped.
Idempotency keys for external calls. The client supplies a key; you store the response against it and return the stored response on a repeat. This is the mechanism in 9.22, and it is what payment providers expect.

[!] Ordering, and the trap in partition keys

Kafka guarantees order within a partition, so ordering per entity means using the entity as the partition key — product ID for catalogue updates. Get this wrong and two updates to the same product can be processed out of order, leaving stale data with no error anywhere.

The trap is that partition key choice also determines parallelism. Key by product ID and you can run as many consumers as partitions. Key by market to guarantee ordering per market, and with four markets you have capped concurrency at four consumers regardless of how many you deploy. This is a common and expensive mistake: ordering guarantees and throughput are bought with the same currency.

[retail] Version stamps as an ordering escape hatch

Where ordering cannot be guaranteed, make out-of-order arrival detectable. Every product.updated event carries the source row's updated_at, and the indexer refuses to overwrite a record whose stored timestamp is newer. Late messages are dropped rather than applied. This is strictly more robust than relying on ordering, because it survives replays, backfills, and the day someone changes the partition key.

16.9 Retries, backoff, and the retry storm

Retrying is the most obvious response to failure and the easiest way to convert a small problem into an outage.

[!] The retry storm

A service slows down under load. Callers time out and retry. Retries add load, so it slows further, so more calls time out and retry. Load increases precisely when the service can least handle it, and a service that was struggling is now unreachable. Retries turned a degradation into an outage. Worse, when the service recovers it is immediately flattened by the backlog of retries waiting to fire.

[+] Four rules that make retries safe

Only retry what is retryable. A 503 or a timeout, yes. A 400 or a 422 will fail identically every time (9.5) — retrying a validation error is pure waste.
Exponential backoff. 1s, 2s, 4s, 8s. Give the dependency room to recover instead of hammering it on a fixed interval.
Jitter, which is not optional. Without randomness, every client that failed at the same moment retries at the same moment, and you have built a synchronised thundering herd. Jitter is one line of code and it is the difference between a spike and a smear.
A retry budget. Cap total attempts and cap retries as a fraction of overall traffic. If more than a few percent of requests are retries, the system is unhealthy and should shed load rather than amplify it.

[!] Retries multiply down a chain

Three services deep, each retrying three times, means the bottom service sees up to twenty-seven attempts for one user request. Retry at one layer — usually the outermost, closest to the user, where you know whether it is still worth doing — and let the inner layers fail fast. Independent retry logic at every layer is a distributed amplifier.

[retail] LLM calls need different retry rules

Retrying a provider call is not like retrying a database read: a failed generation may have consumed thousands of tokens you are still billed for, and a retry costs again. Three attempts against a slow, expensive endpoint can also blow the user's latency budget entirely. So: retry twice at most, respect the provider's Retry-After header rather than your own backoff schedule, and treat a rate limit as a signal to fail over to another provider (16.14) rather than to wait. The spend controls in 8.22 are what stop a retry loop becoming an invoice.

16.10 Timeouts, circuit breakers, bulkheads

Retries handle a dependency that is briefly unwell. These three handle one that is properly broken, and their shared purpose is to stop its failure becoming yours.

[def] Timeouts come first, and are the one people forget

A call without a timeout waits indefinitely. Under load, every request handler ends up parked on the same dead dependency, the connection pool fills, and a service with no bug of its own becomes unavailable. Every network call needs an explicit timeout, and many client libraries default to none. Set it from the caller's budget, not the callee's average: if the user-facing p99 target is 200 ms, a 30-second timeout is meaningless because the user left long ago.

[def] The circuit breaker

A wrapper that counts failures and, past a threshold, stops making the call at all. Three states:

Closed — normal. Calls pass through, failures are counted.
Open — the threshold was breached. Calls fail immediately without touching the network, for a cooldown period.
Half-open — after the cooldown, let a single trial call through. Success closes the circuit; failure re-opens it for another cooldown.

The point is that failing instantly is better than failing slowly. It frees the caller's threads, and it gives the struggling dependency the one thing it needs in order to recover: less traffic.

[def] The bulkhead

Named after ship compartments: isolate resources per dependency so one flood does not sink everything. If the orchestration service has a single connection pool of 100 shared across the LLM, vector search and the catalogue, a slow LLM consumes all 100 and search dies with it. Give each dependency its own bounded pool — say 40, 40 and 20 — and a slow LLM exhausts only its own allocation. Search keeps working, and you get a partial outage instead of a total one.

[+] The breaker is only half the design

Opening a circuit is easy. The real question is what you serve while it is open, and that is a product decision as much as an engineering one. Options, in descending order of niceness: serve a cached or stale result; serve a degraded result from a cheaper path; serve a partial response with the failed section omitted; or return a clear error. An interview answer that says "circuit breaker" and stops has done the easy half.

[retail] Four dependencies, four different fallbacks

Degradation policy per dependency
Dependency down	What the user gets
Reranker (5.x)	Vector search results, unreranked. Slightly worse ordering; most users never notice.
Vector search (chapter 4)	Fall back to Postgres keyword search. Materially worse results, but a working search box.
LLM provider (chapter 6)	Fail over to the secondary provider (16.14). If both are down, show retrieved documents with no generated summary.
Catalogue Postgres	No useful fallback. This is the system of record; return an honest error. Not everything degrades gracefully, and pretending otherwise serves wrong prices.

16.11 Distributed transactions and sagas

This is the hardest consequence of database-per-service, and the one most likely to be probed in an interview, because every honest answer involves giving something up.

[!] Why two-phase commit is not the answer

2PC has a coordinator ask every participant to prepare, then tells them all to commit. It does deliver atomicity. It also holds locks across the network for the duration, and if the coordinator dies after the prepare phase, participants are left holding locks with no way to decide. It couples the availability of every participant together — precisely the thing the split was meant to avoid. Most modern stacks, including everything in this course, do not offer it across services. Naming 2PC in an interview is fine; recommending it is not.

[def] The saga

Replace one distributed transaction with a sequence of local transactions, each paired with a compensating action that semantically undoes it. If step four fails, run the compensations for three, two and one in reverse. You trade atomicity for availability, and the price is real: the system now passes through intermediate states that were impossible inside a transaction, and it has to be correct in every one of them.

Orchestration versus choreography
	Orchestrated	Choreographed
How it works	A coordinator calls each step in turn and tracks progress	Each service reacts to the previous one's event and emits its own
Reading the flow	Written down in one place	Implicit across N services; nobody can see the whole thing
Coupling	The coordinator knows every participant	Services know only their own events
Debugging a stuck flow	Ask the coordinator for its state	Reconstruct it by correlating logs across services
Best for	Business-critical flows with more than three steps	Short flows across genuinely independent teams

[+] Prefer orchestration for anything involving money

Choreography is elegant in diagrams and painful at 3am. When an order is stuck half-completed, "which service is waiting on what" needs to be answerable with one query, not reconstructed from five log streams. Explicit state in a coordinator is worth the coupling for any flow where being stuck is expensive.

[retail] Order placement as a saga, including the awkward step

Reserve stock Catalogue service decrements available stock. Compensation: release the reservation.
Authorise payment Payment service holds the funds. Compensation: void the authorisation.
Create the order Order service writes the record. Compensation: mark it cancelled — not delete it, because something may already have read it.
Send confirmation Notification service emails the customer. Compensation: none exists.

That last step is the one to raise before the interviewer does. You cannot unsend an email, so irreversible steps go last, after everything that can fail already has. If an irreversible step genuinely must happen mid-saga, the compensation is not technical — it is a second email apologising, which is a product decision someone has to actually make and own.

[!] Compensation is not rollback, and the difference is visible to users

A rollback erases history; a compensation adds to it. A refund is not the absence of a charge — the customer sees both lines on their statement, and may have paid a currency conversion fee on each. Stock released after a reservation may have been unavailable to another shopper for those thirty seconds. Sagas leak their intermediate states into the real world, and the design question is not how to hide that but which leaks are acceptable.

16.12 Distributed tracing

With one service, a stack trace tells you what happened. With eight, a request is spread across eight log streams with no shared thread, and tracing is what puts it back together.

[def] Traces, spans, and context propagation

A trace is one request's entire journey, identified by a trace ID. A span is one unit of work within it — a service handling the request, a database query, an LLM call — with a start time, a duration, and a parent span. Together they form a timeline showing where the time actually went. The mechanism is unglamorous: the trace ID travels in request headers, and every service must pass it along. One service that drops the header severs the trace, leaving two disconnected halves with nothing linking them.

[+] Standardise on OpenTelemetry

It is the vendor-neutral standard covering traces, metrics and logs, which is why 14.21 treats them as one concern rather than three. Instrument once against OpenTelemetry and export to whichever backend you use; changing vendors becomes a configuration change instead of re-instrumenting every service. Auto-instrumentation covers FastAPI, HTTP clients and database drivers with very little code, which handles most of the plumbing for free.

[!] Sampling, and the trap inside it

Tracing every request is expensive at volume, so you sample. The trap is that head-based sampling decides at the start of the request, before anyone knows whether it will be interesting — so the one-in-a-thousand failure you actually want is almost certainly not sampled. Tail-based sampling buffers the spans and decides afterwards, letting you keep every error and every slow request while sampling the boring successes at a low rate. It costs more to operate and is nearly always worth it, because the traces worth having are by definition the unusual ones.

[retail] What a trace tells you about a slow search

A search that took 1.4 seconds against a 200 ms budget, broken down by span:

trace 7f3a91 — POST /searchtext

gateway                     1420 ms  [==========================]
  orchestrator              1405 ms  [=========================]
    embed query               38 ms  [=]
    vector search            120 ms  [==]
    rerank                    95 ms  [==]
    llm summarise           1140 ms  [=====================]
      queue wait             890 ms  [================]
      generation             250 ms  [====]

Without the trace, the report is "search is slow" and three teams start investigating in parallel. With it, the answer is immediate and specific: generation was fine, but the request spent 890 ms queued waiting for a GPU slot. That is a capacity or batching problem in the inference tier (chapter 6), not a search problem. Note also that the queue-wait span exists only because someone instrumented it deliberately — time you do not create a span for silently disappears into the parent, and shows up as an unexplained gap.

[+] Three things worth putting on every span in an AI system

Token counts, input and output. This is what makes cost attributable per request rather than per month, and it connects directly to the spend controls in 8.22.
Model and provider name. When latency doubles, the first question is whether the gateway failed over to a slower provider (16.14). Without this attribute you cannot tell.
Retrieved document IDs. When an answer is wrong, the first question is whether retrieval or generation failed. These attributes let the span answer it without reproducing the query.

16.13 Decomposing an AI platform

The source syllabus for this material lists eight AI services: embedding, retrieval, reranking, LLM, recommendation, conversation, model gateway, and orchestration. Drawing all eight as separate boxes is the most common weak answer in an AI design interview.

[!] Apply the 16.1 test to each one honestly

A component earns a network boundary when it needs different hardware, scales on a different axis, or is owned by a different team. Most items on that list fail all three tests. "Reranking service" sounds architectural, but if the reranker is a small cross-encoder running on the same CPUs as the orchestrator, serving the same traffic, maintained by the same people, then making it a service adds a hop, a timeout, a circuit breaker and a deployment — and buys nothing.

Each candidate service, judged against the split criteria
Candidate	Verdict	Why
LLM inference	Service	GPUs, minutes-long startup, and utterly different scaling. The clearest split in the system.
Embedding	Service	GPU or accelerated CPU, and two very different traffic shapes: bulk re-embedding versus per-query.
Vector search	Service	A large resident memory footprint (chapter 4). Scales with corpus size, not user count.
Model gateway	Service	Owns provider credentials and global spend state. Must be shared by every caller to enforce a budget at all (16.14).
Orchestration / retrieval	Service	The thing that composes all the rest. This is where the RAG pipeline lives.
Reranking	Usually a library	Promote it only if it needs its own GPU. A CPU cross-encoder belongs inside the orchestrator.
Prompt service	Library	String assembly. A network call to build a string is pure overhead. Prompt versioning is a storage concern, not a service.
Conversation / memory	Library plus a datastore	This is Redis or Postgres with a thin access layer. A service in front of it is a proxy that adds latency.
Recommendation	Depends entirely	A separate team and its own models, yes. A vector similarity query against the existing index, no.

[+] The shape this lands on

Five services rather than eight, with reranking, prompting, memory and guardrails as libraries inside the orchestrator. Saying this out loud — "these four are modules, not services, and here is why" — is a stronger signal than drawing more boxes. It demonstrates you know what distribution costs, which is exactly what a senior interviewer is listening for.

[retail] One request through the five

Gateway authenticates and routes Token validated, tenant identified, trace ID injected (16.4, 16.12).
Orchestrator receives the query Runs input guardrails in-process (15.11) — a library call, not a hop.
Embedding service turns the query into a vector gRPC, roughly 40 ms, batched with any concurrent queries.
Vector search returns candidates Filtered by tenant at the index level (15.20), roughly 120 ms.
Orchestrator reranks and builds the prompt Both in-process. No network involved.
Model gateway routes to a provider Checks the tenant's spend budget, picks a provider, streams tokens back.

Four network hops for a complete RAG query. The eight-service version would have eight, each with its own p99 tail, and would not answer a single question better.

16.14 The model gateway

Of the five services, this is the one with no equivalent in a conventional backend, and the one interviewers probe hardest. It is worth being able to say precisely why it exists.

[def] What it is, and the one thing that justifies it

A single internal service that every LLM call goes through, presenting one interface over several providers. It holds the API keys, chooses the provider, enforces budgets, and records what was spent.

The justification is not abstraction — it is that spend is global state. A tenant's monthly token budget cannot be enforced by five services each holding their own counter; they would each independently believe the tenant is within budget. Anything that must be counted across all callers has to live somewhere all callers share.

[+] What belongs in it

Credential custody. Provider keys exist in exactly one service, not scattered through every codebase that wants to call a model (15.19).
Routing and failover. Cheap model for classification, strong model for final answers, automatic failover when a provider degrades.
Budget enforcement. Per-tenant and per-feature limits, checked before the call, with the hard stop from 8.22.
Caching. Identical prompts need not be paid for twice. Discussed below, because it is less useful than it sounds.
Usage accounting. Tokens attributed to a tenant and a feature, which is what makes AI cost a line item somebody owns rather than a mystery.
Normalising provider differences. Streaming formats, token-count fields and error taxonomies all differ; callers should not each learn three dialects.

[!] It is a single point of failure, by construction

Every AI feature now depends on it, so it must be more available than anything it fronts: several replicas, no local state beyond caches, and a deployment process more conservative than the services calling it. The failure to avoid is a gateway that grows prompt logic and business rules, because then every prompt change becomes a deployment of the most critical service in the system — the same trap as the API gateway in 16.4, with a bigger blast radius.

[!] Semantic caching, and why the obvious version is dangerous

Exact-match caching on the prompt string is safe and boring: same input, same output, no risk. Semantic caching — embedding the query and reusing the answer from a sufficiently similar past query — sounds like a much bigger win and is where people get hurt.

"What is the return window for shoes?" and "What is the return window for socks?" are highly similar as embeddings and may well have different answers. The similarity threshold is doing safety-critical work, and it has no principled value. Worse, cached answers must be scoped per tenant, or the cache becomes a cross-tenant data leak of exactly the kind 15.20 warns about. Use exact-match caching freely; treat semantic caching as a feature requiring its own evaluation, not a default.

[retail] A routing policy worth defending

Model selection by task, not by preference
Task	Route to	Reasoning
Query intent classification	Small local model	Runs on hardware you already have. Thousands per minute; a frontier model here is indefensible.
Support answer generation	Mid-tier hosted model	Quality matters and a customer is reading it, but it is not a reasoning problem.
Agent planning (chapter 8)	Frontier model	Multi-step tool use is where weaker models fail expensively and invisibly.
Bulk catalogue enrichment	Batch API, overnight	Nobody is waiting. Batch pricing is typically around half, for latency you do not need.

The reason to centralise this is that the policy changes often — new models arrive monthly — and you want it changing in one configuration, not in five codebases.

16.15 Where AI breaks the standard advice

Most microservices guidance was written for request/response CRUD systems. Several of its assumptions simply do not hold for inference workloads, and knowing which ones is a reliable way to sound like someone who has actually run this.

Standard assumptions, and what AI does to them
Usual assumption	What is actually true
Requests take tens of milliseconds	Generation takes seconds. Gateway and proxy timeouts tuned for CRUD will cut off working requests, and streaming (9.17) becomes structural rather than a nicety.
Instances start in seconds	Loading weights takes minutes (14.10, 14.12). Reactive autoscaling is far too slow — by the time a replica is ready the spike is over.
Compute is cheap and roughly uniform	A GPU replica can cost more per hour than a rack of API pods. Idle capacity is a real line item, so scale-to-zero and queueing matter more than usual (14.22).
Identical inputs give identical outputs	Generation is non-deterministic by default. Caching, testing and debugging all assume determinism, and none of them work unchanged.
Health means the process is up	A model server can be up, fast, and producing degraded answers. Liveness probes cannot see quality; only evaluation can (5.33).
Cost scales with request count	Cost scales with tokens. One request with a large retrieved context can cost a hundred times another, so per-request rate limiting does not bound spend.

[+] Queue in front of GPUs, do not autoscale at them

The standard reflex to load is horizontal autoscaling. With multi-minute startup and expensive hardware, the better structure is an explicit queue with bounded workers: requests wait a little instead of failing, batching improves throughput per GPU, and the queue depth becomes the honest scaling signal (14.11) rather than CPU utilisation, which tells you almost nothing about a GPU workload.

[!] The bounded queue needs a limit and a policy

An unbounded queue does not prevent an outage, it disguises one: requests accumulate, latency climbs past the point of usefulness, and users receive answers they stopped waiting for. Cap the depth, reject with a clear error when it is exceeded, and set the cap from the latency budget — if the queue is deeper than what can be served within the timeout, admitting more work is a lie. This is the load-shedding half of 9.19's backpressure discussion, applied to hardware you cannot quickly add.

[retail] Warm pools, because the spike is predictable

Retail traffic is not random: it rises through the morning, peaks in the evening, and explodes on known dates. Weight-loading latency means reacting is hopeless, but predicting is easy. Scale on a schedule ahead of the curve, keep a small warm pool through the trough, and reserve reactive scaling for genuine surprises. This is unfashionable advice in an autoscaling world and it is correct whenever provisioning takes longer than the spike does.

16.16 How the interview is actually scored

A system design interview is not a test of whether you know the right answer. There isn't one. It is a test of how you make decisions when the problem is underspecified, which is what the job is.

[def] What the interviewer is listening for

Do you ask before you build?: The question is deliberately vague. Candidates who start drawing immediately are designing for a problem nobody stated. Five minutes of requirements is not a delay, it is the first thing being scored.
Can you justify each component?: Every box should answer "what happens if I remove this?" A Kafka on the diagram that nobody can motivate is worse than no Kafka, because it invites exactly the question you cannot answer.
Do you name trade-offs unprompted?: "I'd use a cache here, which means accepting staleness up to the TTL; that's fine for product descriptions and not for stock levels." That sentence scores higher than a more sophisticated design presented as free.
Do you know what breaks?: Senior candidates volunteer failure modes. Junior candidates present a happy path and get asked. Reaching the failure discussion first is the strongest single signal.
Do you use numbers?: "That's about 12,000 queries per second, so a single Postgres won't hold it" beats "that could be a lot of traffic." The arithmetic does not need to be precise; it needs to exist (16.18).

[+] A time budget for 45 minutes

Requirements and scope — 5 to 8 minutes Functional, non-functional, and an explicit statement of what you are not building.
Estimation — 5 minutes Traffic, storage, cost. Enough to size things and rule options out.
High-level design — 10 minutes Boxes and arrows. The whole system, shallow, end to end before any depth.
Deep dive — 15 minutes One or two components, usually chosen by the interviewer. This is where the level is decided.
Failure, scale and cost — 5 to 10 minutes What breaks, what you would monitor, what it costs, what you would do differently at ten times the load.

[!] Five ways to lose the room

Designing for a scale nobody asked for. Sharding a database for a system serving 200 internal users signals poor judgement, not ambition.
Naming technologies instead of properties. "I'd use Kafka" is weaker than "I need replay and multiple independent consumers, which points at a log rather than a queue" — then naming Kafka.
Going deep too early. Spending fifteen minutes on the database schema before the system has a shape leaves no time for the parts being assessed.
Silence. Thinking quietly for two minutes reads as being stuck. Narrate the reasoning, including the options you reject.
Defending a bad idea. When the interviewer pushes, they are usually offering information. Absorbing a correction well is a positive signal; digging in is not.

16.17 Requirements, functional and not

The requirements phase has a specific job: turn an open-ended prompt into a bounded problem with numbers attached, so that later decisions have something to be right or wrong about.

[def] Functional versus non-functional

Functional requirements are what the system does: a user can search products, an agent can issue a refund. They determine the components.

Non-functional requirements are how well it must do them: latency, availability, consistency, scale, cost, compliance. These determine the architecture. Two systems with identical features and different latency targets look nothing alike, which is why skipping this section produces a design that cannot be defended.

[+] The questions worth asking, and why each one changes the design

Requirements questions with architectural consequences
Ask	Because the answer changes
How many users, and how many are concurrent?	Everything. This is the first number and all estimation hangs off it.
What latency is acceptable, at p99 rather than average?	Whether you can afford reranking, multi-step agents, or a synchronous chain at all.
How stale may data be?	Whether you can cache, and whether indexing can be asynchronous (16.6).
Read-heavy or write-heavy?	Whether read replicas and caches solve the problem, or whether you need to shard writes.
Is it multi-tenant, and how strict is isolation?	Whether tenancy is a filter, a partition, or a whole separate deployment (16.22).
What is the cost ceiling?	Model choice, GPU count, and whether the design is viable at all.
What happens when the AI is wrong?	Whether you need human approval, confidence thresholds, or a fallback path (15.16).

[+] State the non-goals out loud

"I'm going to assume authentication already exists and not design it, and I'll treat payments as an external provider." This is not dodging; it is scoping, and it buys you the time to go deep where it counts. Interviewers consistently read explicit non-goals as a senior signal, because managing scope is most of the job.

[!] Availability targets are a budget, and most people overspend

Candidates say "99.99%" reflexively. That is 4.3 minutes of downtime per month, which rules out single-region deployment, rolling restarts that drop connections, and most maintenance windows — and it roughly doubles the infrastructure bill. For an internal tool, 99.5% is honest and buys you a much simpler system. Ask what the target is; do not assume the highest one and quietly pay for it.

16.18 Back-of-envelope estimation

Estimation is not about precision. It is about establishing an order of magnitude fast enough that it changes what you design, because the difference between 70 and 70,000 requests per second is the difference between one server and a project.

[def] The numbers worth memorising

You need very few, and rounding aggressively is fine — nobody is checking your long division.

A day is ~100,000 seconds. The real figure is 86,400. Using 100,000 makes the division trivial and errs on the safe side.
Peak is 3 to 5 times average. Daily traffic is never flat; design for peak, not the mean.
One token is about 4 characters, so roughly 750 words per 1,000 tokens.
A modest server handles ~1,000 simple requests per second. A Postgres instance handles thousands of simple reads per second, far fewer complex writes.
Memory beats SSD by ~1000x; SSD beats network by a lot. Enough to reason about where a bottleneck must be.

[retail] Sizing the search platform, step by step

estimationtext

GIVEN     2,000,000 daily active users, ~3 searches each

TRAFFIC   6,000,000 searches/day
          6M / 86,400s               =    69 qps average
          x5 for evening peak        =   347 qps peak

AI SHARE  ~10% of searches trigger a generated summary
          600,000 AI calls/day       =    35 qps at peak

TOKENS    2,500 in (retrieved context dominates) + 300 out
          600k x 2,500               =  1,500M input tokens/day
          600k x   300               =    180M output tokens/day

COST      at $0.15 / $0.60 per 1M tokens
          1,500 x 0.15 + 180 x 0.60  =   $333/day
                                     = ~$10,000/month

Three conclusions fall straight out. 347 qps is not a scaling problem — a handful of API pods handle it, so no sharding, no exotic infrastructure. Input tokens dominate cost by eight to one, so the highest-leverage optimisation is retrieving fewer or shorter chunks, not switching models. And $10,000 a month is a real budget line that someone will question, which is why 16.14's per-tenant accounting exists.

[+] Let the estimate kill options out loud

The point of the arithmetic is to eliminate. "347 peak qps means a single Postgres with a read replica is fine, so I'm not going to shard" is a decision the interviewer can check. Equally: if reranking every result with a cross-encoder costs 40 ms and the p99 budget is 200 ms, that is affordable — but reranking with an LLM call at 800 ms is not, and you have just ruled it out with a number rather than an opinion.

[!] Storage estimates are where people forget the multipliers

Raw data size is the beginning, not the answer. 800 million 256-dimension int8 vectors is about 205 GB of vector data — then add the HNSW graph, which quantization does not shrink (4.6), then replication, then headroom for compaction. The honest figure is routinely two to three times the naive one. The same applies to databases: indexes, write-ahead logs and backups are frequently larger than the table.

[+] Say your assumptions as you use them

"I'll assume 10% of searches trigger generation — tell me if that's wildly off." This invites correction on the input rather than the conclusion, and it means a wrong assumption costs you one sentence instead of the whole design. It also shows you know which variable the answer is sensitive to, which is the actual skill being tested.

16.19 The component vocabulary

A design interview draws from a small set of building blocks. Knowing what each one is for — and what it costs — is what lets you place it deliberately rather than decoratively.

The standard components, and the question each one answers
Component	Solves	Costs
Load balancer	Spreading traffic; removing unhealthy instances	Another hop; sticky sessions if you were careless about state
API gateway	One entry point for auth, rate limiting, routing (16.4)	A single point of failure that must be very available
Cache	Repeated reads of the same data	Staleness, invalidation, and a cold-start cliff after a flush
CDN	Static assets served near the user	Purge latency; useless for personalised responses
Message queue	Decoupling and load levelling (16.6)	Eventual consistency; a dead-letter queue someone must watch
Object storage	Large blobs: documents, images, model weights	Higher latency than disk; per-request cost at volume
Read replica	Read-heavy relational load	Replication lag — a read-after-write can miss its own write
Search index	Text and vector retrieval (chapter 4)	A derived copy that must be kept fresh and can be rebuilt

[+] Caching: the three questions that matter

Candidates add a cache reflexively and then cannot answer the follow-ups. There are only three:

What is the key? If it includes the user ID, hit rates collapse and the cache may be pointless. If it does not, check very carefully that the response is not personalised — this is a classic cross-tenant leak.
How does it get invalidated? TTL is simple and means accepting staleness. Event-driven invalidation is precise and adds a coupling. "Both" is a legitimate answer: short TTL as a safety net, events for correctness.
What happens when it is empty? After a deploy or a flush, every request goes to the origin at once. If the origin cannot survive that, the cache is not an optimisation, it is a dependency.

[!] Consistency, stated in the terms interviewers use

Strong consistency means a read always sees the latest write; eventual means it will, given time. The useful middle term is read-your-own-writes: other users may see stale data, but you always see your own changes. That is usually what a product actually needs, and it is cheaper than strong consistency — often just routing a user's reads to the primary briefly after they write. Naming this specific guarantee, rather than reaching for strong consistency everywhere, is a strong signal.

[retail] Where each one lands in the search platform

CDN for product images. Obvious, high value, no complications.
Cache on embeddings of popular queries. "black running shoes" is embedded thousands of times a day and the vector never changes — a near-perfect cache: immutable value, high repeat rate, no personalisation.
No cache on generated summaries, at least not semantically (16.14).
Queue between catalogue updates and re-embedding. Load levelling for GPU work.
Read replica for the catalogue, with the caveat that stock levels must come from the primary — replication lag on a stock count sells things you do not have.

16.20 Scaling moves, in order

When the interviewer says "now ten times the traffic", there is a conventional order to the answer. Skipping ahead to the exotic moves is the classic overreach.

[+] The ladder, cheapest first

Measure before assuming Which resource is actually saturated? Scaling the wrong tier is the most common waste, and "I'd look at the traces first" (16.12) is a legitimate first answer.
Scale vertically A bigger machine is unglamorous and often correct. It requires no code changes and modern hardware goes a long way.
Add a cache Usually the largest single win per unit of effort, if the read pattern is repetitive.
Scale horizontally More stateless replicas behind the load balancer. Cheap and easy if the service holds no local state, which is the actual design requirement here.
Add read replicas For read-heavy relational load. Accept replication lag, and route the reads that cannot tolerate it to the primary.
Make work asynchronous Anything the user is not waiting for moves behind a queue (16.6).
Shard the data The last resort. It complicates every query, makes cross-shard joins and transactions painful, and rebalancing is a project. Justify it with a number before proposing it.

[!] The bottleneck usually moves rather than disappears

Fix the API tier and the database becomes the constraint. Fix that and it is the network, or the GPU queue, or connection limits. Saying this explicitly — "that removes the API bottleneck, and I'd expect the database connection pool to be next" — demonstrates you have run systems rather than only drawn them.

[retail] Ten times traffic, applied to the estimate

3,470 peak qps instead of 347. The API tier scales horizontally — boring and fine. The vector index is unchanged, because it scales with catalogue size, not query volume; it needs more replicas for throughput, not more memory. Postgres gets read replicas. The genuine problem is the LLM tier: 350 concurrent generations at roughly $100,000 a month, where GPUs cannot be added in seconds (16.15). That is where the answer becomes queueing, smaller models for the easy cases, and asking whether 10% of searches really need generation — an architecture question that turns into a product question, which is the honest place for it to land.

16.21 Scenario: semantic product search

The first of three worked designs. This one is deliberately the most familiar — it is the system built across chapters 3, 4 and 5 — so the focus is on the method: how the phases from 16.16 actually sound when strung together.

Requirements, in about five minutes

What to establish before drawing anything
Question	Answer taken
Scale	2M daily active users, 6M searches/day, 347 peak qps (16.18)
Catalogue	800M products across four markets
Latency	p99 under 300 ms for results; a generated summary may stream after
Freshness	New products searchable within about a minute; price and stock immediately correct
Availability	99.9%. Search degrading is bad; search being wrong about stock is worse
Non-goals	Not designing auth, checkout, or the recommendation engine

[+] The freshness answer is doing real work

"Searchable within a minute, but price and stock immediately correct" is one sentence that determines two decisions. Indexing can be asynchronous, so a queue is justified. But stock cannot be served from the search index, so results must be enriched from the catalogue at read time — which means the index stores IDs, not prices. Getting this backwards produces a system that confidently sells out-of-stock items.

The design

request pathtext

            +-----------+
  user ----> |  gateway  |  auth, rate limit, trace ID
            +-----+-----+
                  |
            +-----v---------+       +-------------------+
            | orchestrator  |-----> | embedding service |  ~40 ms
            +--+---------+--+       +-------------------+
               |         |
               |         |         +-------------------+
               |         +-------> |  vector search    |  ~120 ms
               |                   |  (Qdrant, sharded)|
               |                   +-------------------+
               |
               |  enrich: price, stock, title
               +-----------------> +-------------------+
                                   | catalogue (PG)    |  ~15 ms
                                   |  + read replicas  |
                                   +-------------------+

  ingest path:
    catalogue writes --> Kafka (product.updated) --> embedding
                                                --> index writer --> Qdrant

[+] Why each box is there

Separate embedding service — different hardware, and query embedding batches with concurrent requests (16.13).
Qdrant rather than Postgres pgvector — 800M vectors with a 269 GB working set and per-tenant filtering. This is the one place the scale genuinely forces the choice (4.6).
Kafka on the ingest path — replay, for when the embedding model is upgraded and the whole catalogue must be re-embedded (16.7).
Read replicas for enrichment — except stock, which reads from the primary.

The deep dive you will be pushed into

[!] "What if the vector index and the catalogue disagree?"

They will, constantly — that is the nature of an asynchronous index. The answer is the one from 16.3: the catalogue is the system of record, the index is a derived copy, so you rebuild rather than reconcile. Concretely: a deleted product may still be in the index, so the enrichment step filters out IDs that no longer resolve. That is a deliberate design decision, not a bug — the index is allowed to be wrong, and the read path is responsible for not showing it.

[!] "Your enrichment step is N+1 queries"

Correct, and it is the most likely real bottleneck in this design. Fetching 20 products individually is 20 round trips inside a 300 ms budget. Fix it with a single batched WHERE id = ANY($1) query — one round trip regardless of result count. Volunteering this before being asked is worth more than the fix itself.

[+] The latency budget, added up

Embed 40 ms, search 120 ms, enrich 15 ms, rerank 40 ms, overhead 20 ms — about 235 ms against a 300 ms target. It fits, with little room, which is exactly the observation to make out loud: there is no budget here for an LLM call on the synchronous path. That is why the generated summary streams separately after results are already on screen (9.17), rather than delaying them.

[retail] What breaks, and what the user sees

Failure modes and their degraded paths
Failure	Behaviour
Embedding service down	Fall back to Postgres full-text search. Worse results, working search box.
Qdrant shard down	Replicas cover it (4.11). If a whole shard is lost, results are incomplete but not wrong.
Kafka consumer lag	Index goes stale. Search still works; new products appear late. Alert on consumer lag, not on Kafka being up.
Catalogue primary down	No enrichment, so no prices. Return an error rather than a page of unpriced products.

16.22 Scenario: multi-tenant enterprise RAG

A RAG platform where 500 companies upload their internal documents and ask questions of them. This is the scenario most likely to be failed on a single point, and it is not an architecture point — it is whether you take isolation seriously enough.

Requirements

What changes when the users are other companies
Question	Answer taken
Tenants	500 companies, from 10 to 50,000 employees each — wildly uneven
Corpus	10,000 to 5,000,000 documents per tenant
Isolation	Contractual. One tenant seeing another's document is a company-ending event, not a bug
Latency	p95 under 3 seconds end to end, streaming
Residency	Some EU tenants require data to stay in the EU
Cost	Must be attributable per tenant, because they are billed for it

[!] Say this in the first two minutes

"Before anything else: the failure mode that matters here is cross-tenant leakage, and I want isolation enforced in more than one place." Leading with this reframes every later decision, and it is the difference between a design that happens to be secure and one that was designed to be. An interviewer for a multi-tenant system is waiting to hear it.

The isolation decision

Three isolation models, and what each is honestly worth
Model	Isolation	Cost and fit
Shared everything, filter by tenant_id	Weakest. One missing filter is a breach	Cheapest, and scales to many small tenants
Shared infrastructure, separate collection or schema per tenant	Strong. A bug returns nothing rather than someone else's data	More objects to manage; per-collection overhead bounds tenant count
Dedicated deployment per tenant	Total, and the only real answer for data residency	Expensive. Viable for a handful of large tenants, not 500

[+] The answer is a tiered mix, and saying so is the point

500 uneven tenants do not get one model. Small tenants share a collection with metadata filtering and is_tenant co-location (4.19). Large tenants get their own collection. EU-residency tenants get a separate regional deployment, because residency is a legal boundary and no amount of filtering satisfies it. Recognising that tenancy is a spectrum priced per tenant, rather than one global choice, is what separates a real answer from a textbook one.

[!] Filtering must happen at the index, not in application code

This is the single most important sentence in the scenario. The tenant filter belongs in the vector database's metadata filter, applied as part of the query itself — not as a post-filter over results, and never as application code that remembers to check. Post-filtering means the engine retrieved another tenant's documents and then discarded them, so any bug, any logging statement, any error path that dumps the pre-filter result set is a leak. This is the most common real-world cause of AI data leaks, and 15.20 covers it in full.

Defence in depth

[+] Four independent layers, each assuming the others fail

The token carries the tenant, and nothing else does Tenant identity is derived from the validated JWT at the gateway (9.21). It is never read from a request body, a query parameter, or a header the client can set.
The query builder cannot omit the filter Make it structurally impossible rather than a convention: a repository layer that takes tenant as a required constructor argument, so there is no code path that builds a query without one.
Storage enforces it independently Separate collections for large tenants; Postgres row-level security for shared tables. If the application forgets, the database still refuses.
The response is checked before it leaves Every retrieved chunk carries its tenant ID; assert it matches before the chunks reach the prompt. Cheap, and it catches the case where every layer above has failed.

[!] The leaks that are not retrieval

The cache. A semantic cache keyed on query text alone will serve tenant A's answer to tenant B. Tenant ID must be part of every cache key (16.14).
Fine-tuning. Training one model on all tenants' documents bakes their data into shared weights, where no filter can reach it. This is why RAG is the right architecture for multi-tenant knowledge and fine-tuning is not.
Embeddings themselves. An embedding is derived from the source text and can leak information about it. A shared vector space is usually acceptable; a shared vector index without hard filtering is not.
Logs and traces. Prompts contain retrieved documents. A trace with full prompt text (16.12) is a copy of tenant data in your observability stack, subject to different access controls and a longer retention period than anyone intended.

[retail] The noisy-neighbour problem, which is the other half

Isolation is not only about data. One tenant bulk-uploading 5 million documents must not slow everyone else's queries, and one tenant's runaway agent loop must not consume the shared token budget. Both need the bulkhead thinking from 16.10: per-tenant queues and quotas on ingestion, per-tenant rate limits and spend caps at the model gateway (16.14). Fair-share scheduling on ingestion is unglamorous, and it is what stops the platform's worst day being caused by its best customer.

16.23 Scenario: a scalable LLM gateway

Every team in a company calls LLMs. The mandate is to put one service in front of all of it. This scenario is chosen because it is infrastructure rather than a product — the requirements are about control, and the interesting failures are about state.

Requirements

What the gateway has to be
Question	Answer taken
Callers	40 internal teams, several hundred services
Volume	2,000 requests/second peak, most streaming
Providers	Three hosted vendors plus self-hosted vLLM (chapter 6)
Overhead budget	Under 20 ms added to time-to-first-token
Availability	99.95%. Every AI feature in the company fails with it
Must enforce	Per-team budgets, provider failover, full usage accounting

[!] The constraint that shapes everything: it is a streaming proxy

A normal API gateway receives a request, does work, returns a response. This one holds a connection open for 30 seconds while tokens flow through it. That changes the design fundamentally: connections are long-lived so a rolling deploy must drain rather than cut; memory scales with concurrent requests, not request rate; and the response cannot be buffered, because buffering destroys the streaming experience it exists to provide. Say this early — it is the difference between designing a proxy and designing a request handler.

The design

gateway internalstext

  caller --> [ authn ] --> [ budget check ] --> [ router ] --> provider
                 |               |                   |
              JWT, team       Redis:              health +
              identity        spend counters      cost policy
                                  |
                             async write --> usage log (Kafka)
                                                     |
                                              billing / analytics

[+] Budget checking without adding latency

The naive design reads and writes a spend counter synchronously per request, which puts a database on the critical path of every call. Instead: keep counters in Redis, check them with a single atomic increment (sub-millisecond), and write the authoritative usage record asynchronously to Kafka after the response completes. You cannot know the real token count until the stream ends anyway, so the accounting is necessarily after-the-fact. The check is fast and approximate; the record is slow and exact.

[!] That means budgets can be overshot, and you should say so

Between the check and the accounting, concurrent requests can push a team past its cap — the classic race between a read and a delayed write. The honest answer is that this is acceptable and bounded: overshoot is limited to roughly the number of in-flight requests times the maximum cost of one, which is small relative to a monthly budget. If a hard cap is genuinely required, you must reserve the maximum possible cost up front and refund the difference after — which is correct, slower, and rejects requests that would have fit. Naming that trade-off is the answer; pretending the race does not exist is not.

[+] Failover, and the part people forget

Routing to a healthy provider is easy. Two subtleties are not:

Failing over mid-stream is not transparent. If a provider dies after 200 tokens, you cannot silently switch — the caller has already received those tokens. Either restart the generation and accept duplicated output, or end the stream with an error. There is no third option, and choosing one is a product decision.
Different providers produce different text. Failover changes output quality and format, and prompts tuned for one model may behave oddly on another. Failover is a degraded mode, not an equivalent one, and callers should be able to see which model actually served them (16.12).

[!] Rate limits belong to the provider, not to you

Your gateway's limits are per team. The provider's limits are per account, shared across all your teams — so the gateway must also manage a global budget it does not control. When a provider returns 429, the correct response is to shed or reroute, not to retry into a wall (16.9). This is also the strongest argument for the gateway existing at all: without it, forty teams independently discover the shared rate limit by hitting it.

[retail] Making it survive its own deploys

At 99.95% with every AI feature depending on it, the gateway's own deployment is the biggest risk to its availability. Three things matter: connection draining with a termination grace period longer than the longest generation, so in-flight streams finish; no local state, so any replica can serve any request and losing one costs only its current streams; and stateless configuration reloads, so routing policy changes without a deploy — which is what stops the most-changed thing in the system from requiring a restart of the most critical service.

16.24 Seven more, compactly

The remaining scenarios from the syllabus, each reduced to the thing that actually decides the design. Full treatments would repeat most of 16.21 to 16.23; what varies is the central tension, and that is what an interviewer is probing for.

The deciding question in each remaining scenario
Design a...	The tension that decides it	Where the components come from
AI chatbot	Conversation state. Context grows every turn until it exceeds the window and costs rise with it, so the real design is the truncation and summarisation policy — what you drop, and when. Not the chat UI.	Memory (8.x), streaming (9.17)
Enterprise RAG platform	Ingestion, not retrieval. Candidates design query-time and forget that parsing 50 document formats, chunking well, handling updates and deletions, and recovering from a bad parse is where the engineering actually is.	Chunking (5.x), pipelines (16.6)
AI recommendation engine	Precompute versus real-time. Batch recommendations are cheap and stale; live ones are fresh and expensive. Almost always a hybrid: precomputed candidates, re-ranked live with session context.	Vector similarity (4.x), caching (16.19)
AI customer-support platform	The handoff to a human. The AI path is the easy half; escalation, context transfer, and knowing when to give up are the design. Confidence thresholds (15.15) matter more than model choice.	Agents (chapter 8), approval (15.16)
AI agent platform	Blast radius. Agents take actions, so the design is about bounding them: tool authorisation per tenant, spend caps, loop limits, and an audit trail. This is a security design wearing an architecture costume.	Tool auth (15.16), bounds (5.23, 8.22)
Conversational commerce	Transactional correctness inside a probabilistic system. The model may suggest anything; the order must still be right. Sagas (16.11) and a hard rule that the LLM proposes and deterministic code disposes.	Sagas (16.11), guardrails (15.17)
Enterprise AI automation	Idempotency at scale. Automated workflows retry, and retried side effects in someone's ERP are expensive. Every action needs an idempotency key and a dry-run mode.	Idempotency (16.8), sagas (16.11)

[+] The pattern across all ten

In every one of these, the hard part is not the AI. It is state, consistency, failure handling, or cost — ordinary distributed systems problems, with a non-deterministic and expensive component in the middle making each of them slightly worse. A candidate who spends the whole interview on model selection has answered the least interesting question in the room.

[!] A reusable opening for any of them

"Let me establish scale and latency first, then sketch the whole thing end to end, then go deep wherever you'd like. I'll flag trade-offs as I go, and I'd rather over-explain a decision than have it look arbitrary." This is not a script to recite — it is a structure that prevents the two failure modes in 16.16: designing before asking, and going deep before there is a shape to go deep into.

16.25 Key takeaways

Distribution is a cost you pay to buy something specific. If you cannot name what you are buying, keep the monolith.
The strongest reason to split is different hardware. In AI systems this is usually the only reason you need, and it is unarguable.
A large codebase justifies modules, not networks. A distributed monolith has every cost of distribution and none of the independence.
Get boundaries right in-process first. Moving a boundary inside a monolith is a refactor; moving one between services is a migration with dual writes.
Cut by capability, not by layer or by table. A service whose API is CRUD on one table should not exist.
Database per service, or the split is theatre. A shared schema cannot be changed independently, whatever the deployment diagram says.
Availability multiplies down a synchronous chain; latency adds. Five services at 99.9% give 99.5%, and the tail is what users feel.
Ask "is the user waiting?" That single question decides synchronous versus asynchronous for any given path.
Kafka is a log, RabbitMQ is a queue. Choose Kafka when you need replay or multiple independent consumers; a Postgres table is a legitimate third answer for low volume.
You have at-least-once delivery. Exactly-once does not survive contact with an external side effect, so make the side effect idempotent instead.
Ordering guarantees and throughput are bought with the same currency. The partition key that gives you per-entity ordering also caps your consumer count.
Retries without jitter are a synchronised herd. Retry at one layer only, or three services deep becomes twenty-seven attempts.
Every network call needs an explicit timeout, set from the caller's budget rather than the callee's average. Most client libraries default to none.
A circuit breaker is only half a design. The other half is what you serve while it is open, and that is a product decision.
Sagas trade atomicity for availability, and compensation is not rollback — a refund is visible to the customer in a way a rollback never was.
Put irreversible steps last. You cannot unsend an email, so it goes after everything that can fail already has.
Tracing is the only thing that makes a distributed system debuggable, and tail-based sampling is what keeps the traces you actually need.
Most "AI services" are libraries. Prompt building, memory and guardrails do not earn a network hop. Five services, not eight.
The model gateway exists because spend is global state, not because abstraction is nice.
AI breaks the standard assumptions: multi-minute startup, non-deterministic output, cost driven by tokens rather than requests, and health that cannot be probed.
Queue in front of GPUs rather than autoscaling at them, with a bounded queue and an honest rejection policy.
In a design interview, non-functional requirements determine the architecture. Ask before you draw; state your non-goals out loud.
Use numbers to eliminate options. "347 peak qps, so no sharding" is a decision the interviewer can check.
Enforce tenant isolation at the index, in more than one place. Post-filtering means you already retrieved the other tenant's data.
Volunteer failure modes before being asked. It is the single strongest seniority signal in the room.

[i] Vocabulary check

You should be able to explain: bounded context, distributed monolith, modular monolith, database per service, service discovery, API gateway versus model gateway, command versus event, log versus queue, at-least-once, idempotency key, partition key, exponential backoff with jitter, retry budget, timeout, circuit breaker, bulkhead, graceful degradation, two-phase commit, saga, compensating action, orchestration versus choreography, trace, span, context propagation, tail-based sampling, functional versus non-functional requirements, read-your-own-writes, noisy neighbour, and blast radius.

16.26 Interview drills

Architecture questions rarely have a single right answer, so these model answers are written the way a strong candidate actually talks: a position, the reasoning, and the trade-off named without being asked.

1. When would you not use microservices?

Most of the time, honestly. The split buys independent deployment, scaling and failure, and it costs a network between every component — timeouts, retries, idempotency, tracing, and no transactions across the boundary.

I'd stay with a modular monolith unless something concrete forces the issue: genuinely different hardware, like a GPU model server next to a CRUD API; components scaling on different axes; or separate teams needing separate release cadences. A large codebase is not one of those reasons — that calls for modules, not networks. And I'd want the boundaries right in-process first, because moving one inside a monolith is a refactor, while moving one between services is a migration with dual writes and a backfill.

2. How do you decide where a service boundary goes?

The test I use is: does a typical business change land inside exactly one service? If adding a product field means editing three services, the boundary is wrong no matter how tidy the diagram looks.

So I cut by business capability — "search the catalogue", "take payment" — rather than by technical layer or by database table. Layer-based splits make every feature touch every service. Table-based splits produce services whose entire API is CRUD on one entity, which have no independent reason to exist. The related point is that the same word means different things in different contexts: a "product" to the search index is an ID and a vector, and to the order service it's an ID, a captured price and a quantity. Trying to force one shared Product model across all of them recreates the coupling you split to remove.

3. Two services need the same data. What do you do?

First I'd check whether they actually need the same data or just the same word. Often each needs a different narrow slice, and that's fine.

If it's genuinely shared, the options are: one service owns it and the other calls the API, which is simple but couples availability; or the consumer keeps a derived copy updated by events, which is fast and independent but eventually consistent. What I would not do is let both read the same tables — that's a shared schema, and a shared schema can't be changed independently, which means you no longer have separate services. If I take the copy approach, I'd be explicit that the owner is the system of record and the copy gets rebuilt rather than reconciled when they disagree.

4. Explain the saga pattern, and its cost.

Once each service owns its database you can't have one transaction across them. A saga replaces it with a sequence of local transactions, each with a compensating action that semantically undoes it. Step four fails, you run the compensations for three, two and one in reverse.

The cost is real and worth naming. You've given up atomicity, so the system passes through intermediate states that were previously impossible, and it has to be correct in all of them. And compensation isn't rollback — a rollback erases history, a compensation adds to it. The customer sees both the charge and the refund on their statement. I'd also make sure irreversible steps come last, because you can't unsend a confirmation email. For anything involving money I'd use an orchestrated saga rather than choreography, so that "which step is this order stuck on" is one query rather than a log-correlation exercise at 3am.

5. Your service calls a dependency that has become slow. Walk me through what happens and what you'd do.

If there's no timeout, request handlers pile up waiting, the connection pool fills, and my service becomes unavailable despite having no bug of its own. That's the first thing to fix: every network call gets an explicit timeout, set from my latency budget rather than the dependency's average — a 30-second timeout is meaningless if the user left after two.

Then a circuit breaker, so that once failures cross a threshold I stop calling entirely for a cooldown and fail instantly instead of slowly. That frees my threads and gives the dependency the reduced traffic it needs to recover. Bulkheads matter too — separate connection pools per dependency, so a slow LLM can't consume the capacity that vector search needs. But the important half is deciding what I serve while the circuit is open: stale cache, a degraded path, a partial response, or an honest error. Saying "circuit breaker" and stopping there has only done the easy part.

6. Why can't you have exactly-once delivery?

Because the message system and the side effect are different systems and can't share a transaction. A consumer processes a message, does something, then acknowledges. Crash between the side effect and the acknowledgement and it's redelivered, so the effect happens twice. Acknowledge first instead and a crash loses the work. There's no safe ordering of those two steps.

Kafka does offer exactly-once for read-process-write cycles entirely within Kafka, which is genuine but doesn't extend to charging a card. So I assume at-least-once and make the side effect idempotent. Cheapest first: prefer absolute values to deltas, because setting stock to 12 is repeatable and decrementing isn't. Then upserts on a natural key. Then dedup on a message ID with a unique constraint in the same transaction as the work. Then idempotency keys for external calls.

7. Kafka or RabbitMQ?

They're different data structures. Kafka is a distributed append-only log — consumers track their own offsets, nothing is removed on read, so multiple consumer groups read independently and any of them can rewind. RabbitMQ is a broker: a message is routed to a queue, delivered, acknowledged, gone.

So: Kafka when several independent consumers need the same stream, when I need to replay history to rebuild a derived store, or when I need ordering per key. RabbitMQ when I'm distributing discrete tasks to workers and want rich routing or priorities. For the catalogue pipeline I'd pick Kafka specifically for replay — when the embedding model is upgraded, re-embedding the whole catalogue is an offset reset rather than a bespoke backfill job. Worth noting two things that date older advice: Kafka 4.0 removed ZooKeeper entirely, and RabbitMQ 4.0 removed classic queue mirroring in favour of quorum queues and streams. And if the honest volume is a few thousand messages a day with one consumer, a Postgres table works fine.

8. How many services would you use for a RAG platform?

Fewer than the obvious answer. I'd have five: LLM inference, embedding, vector search, a model gateway, and an orchestration service.

The first three are separate because they need different hardware — GPUs for inference and embedding, a large resident memory footprint for the index. The model gateway exists because spend is global state: per-tenant budgets can't be enforced by five services each keeping their own counter. Everything else people usually draw as a service — prompt building, reranking on CPU, conversation memory, guardrail checks — I'd keep as libraries inside the orchestrator. They're pure CPU, they scale with the same traffic, and no separate team owns them. A network call to build a string buys nothing and adds a failure mode. I'd promote reranking to a service only if it needed its own GPU.

9. What does an AI workload change about standard microservices practice?

Several assumptions stop holding. Requests take seconds rather than milliseconds, so proxy timeouts tuned for CRUD cut off working requests and streaming becomes structural. Instances take minutes to start because of weight loading, so reactive autoscaling arrives after the spike is over.

Compute isn't uniform — a GPU replica can cost more than a rack of API pods, so idle capacity is a budget line. Output is non-deterministic, which breaks the assumptions behind caching and testing. Health checks can't see quality: a model server can be up, fast, and producing worse answers than yesterday. And cost scales with tokens rather than requests, so per-request rate limiting doesn't bound spend at all. Practically, that pushes me toward an explicit bounded queue in front of GPUs instead of autoscaling at them, with queue depth as the scaling signal, plus scheduled warm capacity for predictable peaks.

10. Design a multi-tenant RAG system. What's the first thing you say?

That the failure mode that matters is cross-tenant leakage, and I want isolation enforced in more than one place. Everything else follows from that.

The critical detail is that tenant filtering happens in the vector database's metadata filter as part of the query, not as a post-filter and not in application code that remembers to add it. Post-filtering means the engine retrieved another tenant's documents and then discarded them — so any logging statement or error path that dumps the pre-filter results is a breach. Then layers: tenant identity comes only from the validated token, the repository layer makes it structurally impossible to build a query without a tenant, storage enforces it independently via separate collections or row-level security, and I assert on the way out that every retrieved chunk belongs to the right tenant. I'd also tier it — small tenants share a collection, large ones get their own, and EU-residency tenants get a separate regional deployment, because residency is a legal boundary that filtering doesn't satisfy. And I'd mention the non-retrieval leaks: cache keys missing the tenant, fine-tuning baking data into shared weights, and traces storing full prompts.

11. You're told traffic will be 10x next quarter. What do you do?

Measure first — which resource is actually saturated? Scaling the wrong tier is the most common waste, and traces usually answer it in minutes.

Then the cheap moves in order: vertical scaling, because a bigger machine needs no code changes; a cache if reads repeat; horizontal replicas if the service is stateless; read replicas for read-heavy relational load; move anything the user isn't waiting for behind a queue. Sharding is last, because it complicates every query and rebalancing is a project. For our numbers specifically, 10x takes peak from about 350 to 3,500 requests per second — the API tier just scales out, and the vector index doesn't change at all because it scales with catalogue size, not query volume. The real problem is the LLM tier, where GPUs can't be added in seconds and the cost goes to roughly a hundred thousand a month. That's where I'd push back and ask whether every one of those requests genuinely needs generation.

12. How do you debug a request that's slow in production across eight services?

With distributed tracing, because there's no stack trace across a network. A trace ID is injected at the gateway and propagated in headers through every service, and each unit of work becomes a span with a duration and a parent. The result is a timeline showing where the time actually went.

Two things make this work in practice. Every service must pass the header on — one that drops it severs the trace into two unlinked halves. And sampling has to be tail-based: head-based sampling decides at the start of a request, before anyone knows it's going to be slow, so the traces you most want are the ones you didn't keep. For an AI system I'd also attach token counts, model and provider name, and retrieved document IDs to the spans — that turns "the answer was wrong" into a question I can answer from the trace instead of reproducing the query. And the gaps matter: unexplained time usually means a span nobody instrumented, like queue wait ahead of a GPU.

Where this leaves you

This chapter was about the seams. Every previous one built something — an index, a retrieval pipeline, an agent loop, an API, a deployment — and this one was about what happens when those things have to talk to each other across a network, and how to defend the resulting shape to someone who is deciding whether to hire you.

The recurring theme is worth restating, because it runs against most of what is written about architecture: the good answer is usually the smaller one. Fewer services than the diagram wants. Fewer network hops than the syllabus implies. Fewer nines than the reflex reaches for. Distribution, replication and abstraction are all purchases, and the engineers worth working with are the ones who can say what they bought.

[+] Three chapters still to come

Everything up to here builds systems. The final track measures them: statistics for reading a number honestly, machine learning for the models underneath and alongside the AI stack, and experimentation for the only technique in the course that establishes what your work actually caused.