Chapter 7 · Production serving
Orchestrating Inference with Ray
Chapter 6 built one well-tuned vLLM engine. Real traffic needs a fleet of them, called from code that is mostly waiting — and a fleet whose size changes while it runs. This chapter builds up Ray's tools one scenario at a time, ending with a pipeline that grows and shrinks its worker count as GPU nodes are added and removed underneath it.
[!] Version and scope
Every default and API signature in this chapter was read from the Ray 2.57.0 source rather than from memory or documentation, for the same reason as chapter 6: defaults drift, and a wrong one is invisible until it costs you. Ray is a large project; this chapter covers exactly the slice needed to orchestrate I/O-bound inference calls against a fleet of GPU-serving engines like the ones from chapter 6. It is not a general Ray reference.
7.1 An I/O-bound fan-out
Start from the problem, not from Ray's vocabulary. Every concept in this chapter exists because a plain Python loop stops working at some specific point, and seeing exactly where it breaks is what makes the next tool make sense.
Recall the RAG pipeline from chapter 5 and the vLLM deployment from chapter 6. A single request to your search assistant does not run on one machine doing one thing. It:
- calls an embedding model to vectorise the query,
- searches a vector index,
- calls a reranker on the candidates,
- sends the assembled prompt to a vLLM replica for generation,
- and possibly calls a second vLLM replica for a smaller verification pass.
Each of those steps is a network call to a different service. The code issuing the call spends almost all of its time doing nothing — waiting for a response. That is the definition worth fixing in your head before anything else in this chapter:
[def] I/O-bound
Work whose duration is dominated by waiting on something external — a network response, a disk read — rather than by the CPU computing something. Contrast with compute-bound work, like the matrix multiplications inside the model itself. The GPU work in chapter 6 is compute- and bandwidth-bound. The code that calls that GPU work, sitting in your orchestration layer, is almost entirely I/O-bound. This chapter is about that calling code.
Where a plain loop breaks
Serve one request at a time and each of those five calls happens in sequence. If each takes 100–300 ms, your total latency is the sum of all of them — nothing overlaps, and you are paying wall-clock time for waiting that does no work at all.
embedding = embed(query) # waits ~80ms
candidates = search(embedding) # waits ~40ms
reranked = rerank(query, candidates) # waits ~120ms
answer = generate(prompt, reranked) # waits ~900ms
# total: ~1140ms, almost all of it spent waiting, not computingPython's own asyncio or a thread pool fixes part of this on a single machine: while one request waits for its embedding call, another request's generation call can proceed. That takes you further than most people expect. It runs out of road for three reasons that a single process cannot solve no matter how it is written.
[!] What a single process cannot do
- It cannot hold state that must survive past one request — a connection pool to five vLLM replicas, a rolling count of which replica is least loaded, a cache of recent embeddings — without careful, error-prone manual bookkeeping.
- It cannot use more machines than the one it runs on. Threads and asyncio parallelise I/O within a process; they do not span a fleet of GPU nodes.
- It has no answer for a fleet that changes size while running. When chapter 6's vLLM replicas scale from 2 to 8 nodes overnight, nothing about a hand-rolled asyncio loop knows that happened.
[+] What this chapter builds toward
Ray gives you the same idea as asyncio — do other work while waiting — but spread across many machines, with a way to hold state deliberately, and with a built-in mechanism for growing and shrinking the number of workers as load and available hardware change. Sections 7.2 onward introduce exactly enough of it, in the order a real system needs it, to reach that point.
7.2 Tasks: the first building block
The smallest unit Ray gives you: turn an ordinary function into something that runs elsewhere, and get back a placeholder for its result immediately.
Go back to the sequential pipeline in 7.1. Notice that the embedding call and a search against yesterday's cached index don't actually depend on each other — only the reranker needs both results. A plain loop still runs them one after another, because nothing told Python they could overlap. This is the first and simplest case a task solves.
[def] Task
A function decorated with @ray.remote. Calling .remote() instead of calling the function directly schedules it on some worker process — possibly on another machine — and returns immediately with an ObjectRef, a placeholder for a result that may not exist yet. Call ray.get() on that placeholder when you actually need the value, which blocks until it is ready.
import ray
ray.init()
@ray.remote
def embed(query: str) -> list[float]:
return call_embedding_service(query) # network I/O
@ray.remote
def cached_search(index_snapshot: str) -> list[dict]:
return load_and_search(index_snapshot) # disk + network I/O
# Both calls return instantly with ObjectRefs. Ray runs them concurrently.
embedding_ref = embed.remote(query)
cache_ref = cached_search.remote("2026-08-15")
# Block here, once you actually need both results.
embedding, cached_hits = ray.get([embedding_ref, cache_ref])[+] The pattern: submit everything, collect once
The value of a task is entirely in the gap between .remote() and ray.get(). Everything submitted in that gap runs concurrently, on whatever workers are available — including workers on other machines, which a thread pool can never reach. Submit early, collect late, and let Ray decide where each call actually executes.
[!] ray.get() is where the concurrency ends
A common mistake when learning Ray: calling ray.get() immediately after each .remote() call. That forces the second task to wait for the first to finish before it is even submitted, which reproduces the sequential loop from 7.1 with extra steps and no benefit. Collect the ObjectRefs into a list first, and call ray.get() once, on the list, at the point you actually need every value.
7.3 Where tasks run out of road
Tasks fan work out beautifully. They cannot remember anything between calls, and that becomes a real problem the moment you care how many requests you have in flight at once.
[def] The deployment this chapter assumes
Your vLLM deployment from chapter 6 sits behind a load balancer at one stable URL. Requests go to that URL; the balancer picks a replica. What changes over time is the number of GPUs behind it — replicas are added at peak and removed overnight — and a separate capacity API reports how many are currently serving. Your calling code never sees individual replica addresses and never chooses between them. This is the common arrangement, and it is the one that makes the interesting problem visible.
With a single endpoint, load balancing is not your job. But something else is, and it matters more than people expect: deciding how many requests to have in flight against that endpoint at any moment. Get it wrong in either direction and you lose:
Too few in flight
The fleet scaled up to eight replicas and your code is still sending twenty concurrent requests. GPUs sit idle behind the balancer while you pay for them, and throughput is capped by your own caller rather than by the hardware.
Too many in flight
The fleet dropped to two replicas overnight and you are still sending four hundred. They queue at the balancer, every request's latency climbs, and eventually things time out — the same queueing behaviour chapter 6 described inside one engine, now happening in front of the whole fleet.
[!] A task has no memory
The right number of in-flight requests depends on how many GPUs are currently behind the balancer, and staying within it requires knowing how many requests are in flight right now — a count that changes thousands of times a minute as calls start and finish. Every call to .remote() may run in a fresh worker with no knowledge of any other call, so no task can hold that count. A stateless function can be told the current limit; it cannot be the thing that tracks it.
Before reaching for state, one more tool belongs to tasks: waiting for whichever result arrives first rather than all of them.
[def] ray.wait()
Where ray.get() blocks until everything in the list is ready, ray.wait(refs, num_returns=1) returns as soon as any one is. It hands back two lists — the refs that are ready, and the ones still pending — so you can act on the first result and keep the rest running.
# Fan out independent requests to the one endpoint, handle each as it returns.
pending = [generate.remote(p) for p in prompts]
while pending:
ready, pending = ray.wait(pending, num_returns=1, timeout=30.0)
if not ready:
break # nothing landed in 30s - the fleet is in trouble
handle(ray.get(ready[0])) # start work on this result immediately[retail] Why this matters specifically for I/O-bound fan-out
A request that calls out to five things, as in 7.1, rarely needs to wait for the slowest of the five if that result is optional — a secondary reranking pass, a freshness check against a cache that usually misses. ray.wait() with a timeout lets you take what has arrived and proceed, rather than letting one slow dependency set your entire request's latency. This is the same tail-latency thinking as section 5.35, applied to your own orchestration code rather than to the model.
ray.wait() controls how you consume results. It does nothing about how many requests you were allowed to launch in the first place, nor about noticing that the fleet behind the balancer just doubled. For that, the next section introduces the tool that actually holds state.
7.4 Actors: state that lives somewhere
A task is a function call that happens elsewhere. An actor is a whole Python object that lives elsewhere — a process that stays alive across many calls and remembers whatever you tell it to.
Return to the in-flight problem from 7.3. What it needs is a single long-lived object holding two numbers: how many requests are currently in flight, and how many are currently allowed. Increment on start, decrement on finish, refuse when at the limit. That is an ordinary class in plain Python. An actor is that same class, given its own process and made reachable from anywhere in your Ray cluster.
[def] Actor
A class decorated with @ray.remote. Calling .remote() on the class does not run anything — it starts a dedicated worker process holding one instance of that class, and returns an ActorHandle you use to call its methods. Every method call on that handle runs against the same instance, in the order Ray schedules them, so the object's state accumulates exactly as it would in a normal running program.
@ray.remote
class ConcurrencyGovernor:
"""One endpoint, one shared view of how loaded it currently is."""
def __init__(self, endpoint: str, max_in_flight: int):
self.endpoint = endpoint # stable: never changes
self.max_in_flight = max_in_flight # changes as GPUs are added/removed (7.17)
self.in_flight = 0
def try_acquire(self) -> bool:
# Ordinary Python state, read and written on every call.
if self.in_flight >= self.max_in_flight:
return False
self.in_flight += 1
return True
def release(self) -> None:
self.in_flight -= 1
governor = ConcurrencyGovernor.remote("http://vllm.internal", max_in_flight=40)
if ray.get(governor.try_acquire.remote()):
try:
result = call_vllm("http://vllm.internal", prompt)
finally:
ray.get(governor.release.remote())[+] The one-sentence distinction that matters
Use a task when the work is stateless; use an actor when something must be remembered between calls. Everything else in this chapter follows from that single choice. Reach for an actor only when you actually need memory — wrapping every function in an actor "to be safe" adds process overhead and a serialisation point with no benefit, since Ray already parallelises stateless tasks for free.
[!] Method calls on one actor do not run in parallel
A default actor processes one method call at a time, in the order received, exactly like a single-threaded program. Two calls to try_acquire() from different requests queue up rather than overlapping. Here that is precisely what you want — it is what stops two callers both seeing 39 in flight and both deciding they may proceed — but it means one actor is a throughput ceiling. Section 7.7 covers the exception that matters most for I/O-bound work.
7.5 Named actors
The governor in 7.4 is only useful if every request in your system talks to the same instance of it. A per-process limit of 40 across ten processes is a fleet-wide limit of 400, which is not what anyone intended.
Your API server almost certainly runs as several processes — multiple workers behind a process manager, possibly restarted independently, possibly on different machines. Each one needs to reach the same governor actor, not create its own. Creating the actor once and passing its handle to every process that needs it is fragile: it means threading a reference through your entire startup path, and it breaks the moment a new process starts after the actor already exists.
[def] Named actor
An actor created with a name, retrievable from anywhere in the cluster with ray.get_actor(name) — no handle-passing required. Names live in a namespace, a logical grouping that keeps actors from different applications sharing one cluster from colliding on the same name.
# Process A: creates the governor once, at startup.
governor = ConcurrencyGovernor.options(
name="vllm_governor",
namespace="inference",
get_if_exists=True, # a second startup racing this one joins, doesn't duplicate
).remote("http://vllm.internal", max_in_flight=40)
# Process B, C, D: anywhere else in the cluster, no handle was ever passed.
governor = ray.get_actor("vllm_governor", namespace="inference")
admitted = ray.get(governor.try_acquire.remote())[!] get_if_exists is not optional in a real startup path
Without it, two processes starting at nearly the same moment can both attempt to create an actor with the same name, and one raises an error instead of quietly joining the other. get_if_exists=True makes actor creation idempotent: if the named actor already exists, you get a handle to it instead of a collision. Any service that might restart or scale to multiple replicas should treat this as the default, not an edge case worth handling later.
[+] Naming turns an actor into a piece of infrastructure
This is the shift worth noticing: an unnamed actor is a value your code happens to hold, tied to whoever created it. A named actor is closer to a small service — a thing that exists at a known address, that any part of your system can find, whose lifetime is independent of the process that created it. Section 7.6 covers exactly how independent that lifetime can be.
7.6 Actor options and lifetimes
A handful of settings on .options() decide whether an actor survives the process that created it, how it recovers from a crash, and what resources it reserves. All four matter for a governor that is meant to run for the lifetime of your service, not the lifetime of one script.
- lifetime
- None (default) ties the actor to its creating process: when that process exits, the actor is torn down too. "detached" keeps it running independently until explicitly killed — what a named infrastructure actor like the governor needs.
- max_restarts
- How many times Ray restarts the actor if its process crashes. 0 (default) means no restart. -1 means unlimited.
- max_task_retries
- How many times a specific method call is retried on failure, independent of actor restarts. Can be set per-method with @ray.method(max_task_retries=N) to make one risky method more resilient than the rest of the class.
- num_cpus / num_gpus
- Resources reserved for the actor's process. Both accept fractions — num_gpus=0.25 is legal and meaningful, covered below.
governor = ConcurrencyGovernor.options(
name="vllm_governor",
namespace="inference",
get_if_exists=True,
lifetime="detached", # outlives the process that created it
max_restarts=-1, # a crashed governor comes back, state resets to __init__
num_cpus=0, # it counts integers, it does not compute - see below
).remote("http://vllm.internal", max_in_flight=40)[!] A restarted actor loses its state
max_restarts brings the process back and re-runs __init__. It does not remember what self.in_flight held before the crash. For a governor that just rebuilds its picture from the next few requests, this is a fine default. For state that must not be lost — a count that has to be exactly right — the actor needs to persist it externally (a database, an object store entry) and reload it in __init__. Restarts give you availability, not durability; confusing the two is a common and costly mistake.
[+] num_gpus=0 for an I/O-bound actor is a deliberate, important choice
The governor never runs a model, and never even makes the HTTP call — it tracks two integers and answers yes or no. Left at its default, Ray still reserves a full CPU for it, which is harmless but wasteful at scale. State it explicitly: num_cpus=0 for a lightweight coordinator that is genuinely idle between calls, or a fractional value like num_gpus=0.25 for an actor that needs a GPU present but only uses a quarter of one — four such actors can then share a single GPU. This is the resource-accounting half of recognising I/O-bound work: it tells Ray's scheduler the truth about what the actor actually needs, which is what makes packing many lightweight actors onto few machines possible.
7.7 Async actors for I/O concurrency
Section 7.4 warned that one actor processes calls one at a time. For an I/O-bound actor — the entire category this chapter is about — that limitation is usually unnecessary, and Ray removes it automatically the moment your methods are written with async def.
Think about what "one at a time" actually costs the governor. Each call to try_acquire() does almost no work — compare two integers — so serialising those calls barely matters. But it is often convenient to let the same actor own the HTTP session and make the call itself, so acquiring, calling and releasing are one atomic method that cannot leak a slot when a caller crashes. Now "one at a time" means the whole actor stalls on every single generation call, which defeats the entire point of building it.
[def] Async actor
An actor whose methods are declared async def. Ray detects this automatically at actor creation and runs the actor's methods on an internal asyncio event loop instead of one at a time. Multiple in-flight calls can now be concurrently awaiting their I/O within the same single process, exactly like an asyncio web server — while still being one actor with one consistent view of its own state.
@ray.remote
class VLLMGateway:
def __init__(self, endpoint: str, max_in_flight: int):
self.endpoint = endpoint # one stable URL behind the load balancer
self.max_in_flight = max_in_flight
self.in_flight = 0
self.session = aiohttp.ClientSession()
def set_max_in_flight(self, n: int) -> None:
self.max_in_flight = n # updated as GPUs come and go (7.17)
async def generate(self, prompt: str) -> str:
if self.in_flight >= self.max_in_flight:
raise AtCapacity(self.in_flight)
self.in_flight += 1
try:
async with self.session.post(f"{self.endpoint}/generate",
json={"prompt": prompt}) as r:
return await r.json()
finally:
self.in_flight -= 1 # released even if the call raises
gateway = VLLMGateway.options(
name="vllm_gateway", namespace="inference", get_if_exists=True,
lifetime="detached", max_concurrency=200,
).remote("http://vllm.internal", max_in_flight=40)
# Many callers can now be in-flight through this ONE actor simultaneously.
result = await gateway.generate.remote(prompt)[+] max_concurrency is the ceiling on simultaneous awaits
This is the number to reason about deliberately for an I/O-bound actor, in exactly the same spirit as max-num-seqs from chapter 6: it caps how many method calls can be genuinely in flight inside this one process at once. Too low and requests queue behind a limit with spare capacity sitting unused. Too high and you risk exhausting connections or memory on whatever the actor is calling out to. Size it from the same place you'd size a connection pool — the concurrency limit of whatever is on the other end of the call.
[!] This does not add parallelism to compute
An async actor gives you concurrency during waiting, not more CPU. If a method does real computation instead of awaiting I/O, that computation still runs on the one thread backing the actor's event loop and blocks every other in-flight call while it runs. Async actors are the right tool specifically because the router's work is awaiting network calls — the moment a method needs to actually compute something heavy, that work belongs in a task or a separate actor, not inline in an async method.
7.8 Actor pools
One async router actor handles enormous concurrency for cheap work like routing decisions. Some I/O-bound work genuinely needs many separate processes — and managing a list of actor handles by hand gets tedious fast.
Consider the embedding calls from the RAG pipeline in 7.1. Unlike the router, an embedding client typically holds its own connection to an embedding service and you want several of them running — not for CPU parallelism, but because a single connection has its own concurrency ceiling, and one process's async event loop can only be pushed so far before scheduling overhead starts to show. Ten actors instead of one lets you spread a burst of embedding calls across ten independent connections.
You could create ten actors and juggle round-robin dispatch yourself. Ray has a small utility that does exactly this.
[def] ActorPool
ray.util.ActorPool wraps a list of actor handles and gives you .map() and .submit() methods that hand each unit of work to whichever actor in the pool is free next. It is a thin convenience over the pattern of tracking idle actors yourself — useful precisely when the actors are interchangeable, stateless from the caller's point of view, and you want to push a stream of work through however many of them exist.
from ray.util import ActorPool
@ray.remote
class EmbeddingClient:
def __init__(self, endpoint: str):
self.endpoint = endpoint
def embed(self, text: str) -> list[float]:
return call_embedding_service(self.endpoint, text)
clients = [EmbeddingClient.remote(EMBED_URL) for _ in range(10)]
pool = ActorPool(clients)
# Feeds all 500 texts through whichever of the 10 actors frees up next.
embeddings = list(pool.map(lambda actor, text: actor.embed.remote(text), texts_500))[+] Pool size versus concurrency inside each actor
Notice the two dials this gives you, which stack rather than substitute for each other. Pool size is how many processes share the load; async max_concurrency from 7.7 is how much each process can hold in flight. Ten actors at concurrency 20 gives you 200 simultaneous calls spread across ten connections — useful when the bottleneck is per-connection throughput rather than the calling process itself.
[!] A pool has a fixed size until you touch it
ActorPool does not grow or shrink itself in response to load — it distributes work across exactly the actors handed to it at construction. If your embedding traffic doubles at peak, this pool stays at ten unless something adds more actors to it. That is the exact gap section 7.17 closes, once the chapter has built up the pieces needed to close it properly.
7.9 Placement groups
Everything so far assumes Ray puts actors wherever it likes. Sometimes where an actor lands matters as much as whether it runs at all — and chapter 6 already gave you the example that proves it.
Recall tensor parallelism from section 6.14: a 31B model sharded across four GPUs, with every layer synchronising across all four on every forward pass. That synchronisation runs over NVLink if the four GPUs share a node, and over a far slower network link if they don't. If you asked Ray to simply start four GPU worker actors for one TP group, it is entirely free to place them on four different nodes, which would make your model usably correct and unusably slow. This is not a hypothetical — it is the default behaviour if you don't say otherwise.
[def] Placement group
A reservation of resources across the cluster, requested as a group, with a strategy controlling how the bundles are allowed to spread. Actors and tasks are then scheduled into a specific bundle inside that group, guaranteeing they land where the strategy promised — instead of wherever the scheduler otherwise would have put them.
| Strategy | Guarantee | Use for |
|---|---|---|
| STRICT_PACK | All bundles on one node, or the group fails to schedule | A TP=4 shard group that needs NVLink. Correctness depends on this, so fail loudly rather than silently degrade. |
| PACK | Prefers one node, spills to others if it must | Want locality for performance, but a slower fallback beats failing to schedule. |
| SPREAD | Prefers separate nodes, packs if it must | Replica actors that should survive one node failing, but availability matters more than the guarantee. |
| STRICT_SPREAD | Every bundle on a different node, or fails | A fault-tolerant gateway pair where two instances on the same node defeats the purpose. |
from ray.util.placement_group import placement_group
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
pg = placement_group(
bundles=[{"GPU": 1, "CPU": 4} for _ in range(4)],
strategy="STRICT_PACK",
)
ray.get(pg.ready()) # blocks until the whole group can be satisfied
workers = [
VLLMShardWorker.options(
scheduling_strategy=PlacementGroupSchedulingStrategy(
placement_group=pg, placement_group_bundle_index=i
)
).remote(shard_rank=i)
for i in range(4)
][!] STRICT_PACK can fail to schedule at all
Requesting four GPUs strictly on one node fails outright if no single node in the cluster has four free GPUs — even if forty are free in total, scattered across ten nodes. This is the trade the strategy makes deliberately: a loud failure at request time is far better than a silent, slow placement discovered only once the model is running and NVLink didn't happen. Size your nodes and your placement groups together, and treat a scheduling failure here as the system correctly refusing a bad configuration rather than a bug.
[retail] Where SPREAD matters for the gateway
Run two replicas of the named gateway from 7.5 — one primary, one standby — and request SPREAD across two bundles. If the node hosting the primary goes down, the standby is on different hardware entirely and keeps answering. Requesting no placement strategy at all leaves this to chance: Ray might already put them on separate nodes, or might not, and you would only find out during an outage.
7.10 What to use when
Six sections of motivation, collected into the table you'll actually reach for later. Read it as a decision sequence, top to bottom, not a menu.
| Question | If yes |
|---|---|
| Is the work stateless — same inputs, no memory of past calls? | Task. Submit with .remote(), collect with ray.get() once you need every result. |
| Do you only need the first of several results, and the rest are disposable? | ray.wait() instead of ray.get(), with a timeout. |
| Must something be remembered across many separate calls? | Actor. A class holding state, called through its handle. |
| Do multiple independent processes need to reach the same instance? | Named actor with get_if_exists=True, found via ray.get_actor(). |
| Should that instance outlive the process that created it? | lifetime="detached", with max_restarts set deliberately. |
| Is the actor's work mostly awaiting network or disk I/O? | Async actor (async def methods) with an explicit max_concurrency. |
| Do you need many interchangeable workers sharing a stream of tasks? | ActorPool — but remember it is fixed-size until you resize it. |
| Does correctness or performance depend on where actors land relative to each other? | Placement group with the strategy matching the actual guarantee you need. |
[+] The order these questions get asked in a real system
In practice you rarely pick one primitive in isolation. You start with "is this stateless" because it's the cheapest question, and each subsequent yes narrows toward a specific combination. A production inference orchestrator asks all eight questions against different pieces of itself simultaneously — which is exactly the scenario the next section works through.
7.11 A scenario needing all of it
One worked example, using every primitive from this chapter together, because that is how they actually appear in a real system. This is the RAG pipeline from 7.1, fully built out.
The request: embed a query, search a cache, rerank, generate against a chapter-6 vLLM fleet, and do it resiliently enough to survive a node dying mid-flight.
| Component | Primitive | Why this one |
|---|---|---|
| Embedding + cache lookup | Tasks, submitted together | Stateless, independent of each other — the exact case from 7.2. |
| Reranker call, with a 150ms timeout fallback | ray.wait() | If the reranker is slow, proceed with unranked candidates rather than stall the whole request — 7.3. |
| Fixed vLLM endpoint, concurrency governed | Named, detached actor (async) | Must be found by every API process, must survive restarts, must handle many concurrent in-flight calls against the one endpoint — 7.5, 7.6, 7.7. |
| A pool of embedding-service clients | ActorPool of 10 | Spreads a burst of embedding calls over 10 connections rather than one — 7.8. |
| Two gateway replicas, for failover | Placement group, SPREAD | Guarantees the standby isn't on the same node as the primary — 7.9. |
gateway = ray.get_actor("vllm_gateway", namespace="inference")
embed_pool = ray.get_actor("embed_pool", namespace="inference")
async def handle_request(query: str) -> str:
# Tasks: independent work, submitted together (7.2)
embed_ref = embed_pool.submit_one.remote(query)
cache_ref = cached_search.remote(query)
embedding, cached_hits = await asyncio.gather(embed_ref, cache_ref)
candidates = search_index(embedding, cached_hits)
# ray.wait: take the reranker's result if it arrives in time,
# proceed without it otherwise (7.3)
rerank_ref = rerank.remote(query, candidates)
ready, pending = ray.wait([rerank_ref], timeout=0.15)
ranked = ray.get(ready[0]) if ready else candidates
# Named async actor: one stable endpoint, concurrency governed (7.5, 7.7)
answer = await gateway.generate.remote(build_prompt(query, ranked))
return answer[+] Why no single primitive could have done this alone
Tasks alone can't remember how many requests are currently in flight. An actor alone can't fan the embedding and cache lookups out concurrently the way two independent tasks do. A synchronous actor can't hold hundreds of concurrent generation calls in flight against one endpoint. An unnamed actor can't be found by the other three API processes handling other requests. This is the realistic shape of an inference orchestration layer: several small, specific choices, each answering one of the questions from 7.10, composed together rather than one tool doing everything.
7.12 The problem: GPUs come and go
Everything so far assumes a fixed set of vLLM replicas and a fixed set of caller actors. Neither is true in production, and the mismatch between the two is where this chapter's hardest problem actually lives.
Chapter 6 ended by sizing a fleet for a 10-million-request batch job. In a live service, the picture is more fluid than that: GPU nodes are added when a spot-capacity deal appears, removed when a scheduled job needs the hardware back, or lost outright when hardware fails. Chapter 6's vLLM deployment might run on 2 replicas at 2am and 8 at peak lunchtime traffic, and neither number is fixed at deploy time.
[!] Two separate resizing problems, easy to conflate
There are genuinely two different things that need to resize, and they resize for different reasons on different timescales. Conflating them is the single most common confusion people have with Ray in production, so it's worth naming them precisely before writing any code.
| What resizes | Driven by | |
|---|---|---|
| The GPU fleet | Number of vLLM replicas (chapter 6), and the nodes hosting them | Traffic volume, hardware availability, cost policy |
| The caller layer | The governor's in-flight limit, the size of the embedding ActorPool from 7.8 | How many GPU replicas currently exist to call, and how much I/O-bound work is queued |
The second follows the first: if you have 8 vLLM replicas rather than 2, a governor sized to admit "whatever was correct for 2 replicas" needs to know that, and an embedding pool sized for 2 replicas' worth of downstream traffic may now be the bottleneck. This section and the next three build up exactly enough Ray autoscaling machinery to keep both layers honest as the other one changes — without a human updating a config file every time.
7.13 The cluster autoscaler
The layer that adds and removes machines. It knows nothing about vLLM, replicas, or requests — only about whether the actors and tasks people are asking for fit on the nodes that currently exist.
[def] The Ray cluster autoscaler
A process watching resource demand — pending tasks and actors that can't be scheduled because no node has the CPUs, GPUs or memory they asked for — against resource supply, the nodes currently in the cluster. When demand exceeds supply, it asks the underlying infrastructure (Kubernetes, EC2, whatever the cluster runs on) for more nodes. When nodes sit idle past a timeout, it releases them. This is infrastructure plumbing, not an inference concept — it would behave identically whether your actors ran language models or spreadsheets.
Most of the time this runs invisibly: request more GPU actors than fit, wait, and they get scheduled once new nodes join. Occasionally you know demand is coming before Ray does — a scheduled batch job about to start, a launch you're bracing for — and want capacity to exist before the first task asks for it rather than discovering the need reactively.
from ray.autoscaler.sdk import request_resources
# "I am about to need this much, regardless of what's pending right now."
request_resources(resources_list=[{"GPU": 1}] * 32)[!] This is the only autoscaler that touches real hardware
Nothing in the rest of this chapter provisions or releases a GPU node. The cluster autoscaler is the single layer that talks to the cloud provider or the Kubernetes scheduler underneath Ray. Everything in 7.14 onward operates inside the capacity this layer provides — deciding how many replicas or actors to run given the nodes that exist, not deciding whether the nodes themselves exist. Confusing "no replica available" with "no node available" sends you debugging the wrong layer.
7.14 The Serve replica autoscaler
A completely different layer sitting above the cluster autoscaler: Ray Serve decides how many replicas of a deployment should exist, given the traffic it is currently seeing, then asks the cluster autoscaler for the nodes to host them if none exist yet.
This is the layer that matters for wrapping vLLM. A Serve deployment wraps each vLLM engine as a replica; the autoscaler watches how loaded those replicas are and grows or shrinks the replica count on its own.
- target_ongoing_requests
- The number of requests Serve tries to keep in flight per replica, on average. Default 2. Serve compares real load to this target and moves the replica count toward whatever keeps each replica near it.
- min_replicas / max_replicas
- The floor and ceiling. Defaults are 1 and 100 — both almost always worth setting explicitly for a GPU deployment, where 100 replicas may be a fantasy and 1 may be a single point of failure.
- initial_replicas
- How many replicas exist the moment the deployment starts, before any traffic has been observed. Defaults to min_replicas if unset.
- upscale_delay_s / downscale_delay_s
- How long load must stay above or below target before Serve acts. Defaults 30s and 600s — the tenfold gap is deliberate, and 7.15 is entirely about why.
from ray import serve
@serve.deployment(
autoscaling_config={
"min_replicas": 2,
"max_replicas": 8,
"target_ongoing_requests": 4,
"upscale_delay_s": 30,
"downscale_delay_s": 600,
},
ray_actor_options={"num_gpus": 4}, # each replica: one TP=4 vLLM engine
)
class VLLMReplica:
def __init__(self):
self.engine = load_vllm_engine() # the chapter 6 configuration
async def __call__(self, request) -> str:
return await self.engine.generate(request)[+] target_ongoing_requests is a queueing-theory dial, not an arbitrary number
Set it too high and requests queue behind a replica that's already busy, inflating tail latency exactly like an oversized max-num-seqs in chapter 6. Set it too low and Serve adds replicas — each one a full GPU node — for load a single replica could comfortably absorb. The right value tracks how many concurrent requests one vLLM replica can serve well, which chapter 6 already taught you to compute: it's the concurrency ceiling from the KV-cache arithmetic in section 6.5, not a number to guess at.
[!] This autoscaler does not provision hardware
Asking Serve to scale from 2 replicas to 8, each needing 4 GPUs, is a request for 24 more GPUs to exist. If the cluster autoscaler from 7.13 can't get them — a spot quota, a capacity limit — the new replicas sit pending indefinitely and Serve's decision to scale up accomplishes nothing except a longer queue. The two autoscalers must be sized with the same ceiling in mind: max_replicas here should not promise more GPUs than your cluster autoscaler can actually obtain.
7.15 Why the delays are asymmetric
30 seconds to scale up, 10 minutes to scale down. That gap is not a conservative default someone forgot to tune — it's the correct shape for GPU-backed inference specifically, and understanding why prevents the most common autoscaling mistake.
[!] Flapping: the failure this prevents
Imagine symmetric delays — scale down as eagerly as up. A brief lull in traffic triggers a scale-down; traffic resumes seconds later; Serve scales back up. Each cycle costs real time and money: a new vLLM replica means starting a process, loading 48 –58 GiB of weights from disk, and capturing CUDA graphs, which chapter 6 noted costs tens of seconds all on its own. A fleet that flaps between 4 and 6 replicas every few minutes is doing that expensive startup dance constantly while never being stably sized for anything.
The asymmetry is the fix. Scaling up quickly protects users from a real traffic spike — waiting 10 minutes to add capacity while requests queue is a bad experience nobody would choose. Scaling down slowly protects the fleet from reacting to noise: real traffic dips are usually brief, and a 10-minute delay means only a sustained drop actually triggers a removal, by which point it's clearly not noise.
| Traffic shape | upscale_delay_s | downscale_delay_s |
|---|---|---|
| Bursty, latency-sensitive (interactive chat) | Low (15–30s) | High (600–900s) |
| Smooth, predictable diurnal pattern | Moderate (60s) | Moderate (300s) — you can react faster because the pattern is known |
| Batch job feeding a queue (chapter 6's 10M-request job) | Autoscaling largely irrelevant — size for the whole job up front | Same |
[def] min_replicas=0 and downscale_to_zero_delay_s
Serve allows scaling all the way to zero replicas when idle, and lets the 1→0 transition use a separate, typically longer delay than ordinary downscaling — downscale_to_zero_delay_s, falling back to downscale_delay_s if unset. This is the sharpest version of the cost-versus-latency trade in the chapter: zero replicas costs nothing while idle, but the next request pays the entire cold-start cost — process launch, weight loading, CUDA graph capture — that chapter 6 measured in tens of seconds. Use it for genuinely intermittent workloads, such as a rarely used internal tool. Never use it for anything a real user is waiting on interactively.
[+] The general principle behind the specific numbers
Set the up-delay to roughly how long you're willing to let requests queue before more capacity should already be arriving. Set the down-delay comfortably longer than your normal traffic noise, so you never pay a replica's full startup cost to serve a dip that would have passed on its own. When in doubt, keep the asymmetry — scaling down too slowly wastes GPU-hours, which is a bounded and visible cost; scaling down too aggressively causes flapping, which is much harder to notice and diagnose after the fact.
7.16 num_replicas="auto"
Before hand-tuning every field from 7.14, know that Ray Serve has a one-line shortcut that picks sensible defaults for you — and knowing what it actually sets is more useful than treating it as a black box.
@serve.deployment(num_replicas="auto")
class VLLMReplica:
...[def] What "auto" actually resolves to
Setting num_replicas="auto" is equivalent to writing out AutoscalingConfig's own defaults explicitly: target_ongoing_requests=2, min_replicas=1, max_replicas=100, plus a sensible per-replica concurrency cap. It is not magic — it is the same config from 7.14 with values chosen to be reasonable for a typical web service, not for a GPU-bound vLLM replica specifically.
[!] The defaults inside "auto" are wrong for vLLM specifically
max_replicas=100 assumes cheap, plentiful replicas — true for a stateless API handler, false for a GPU engine where each replica is an entire node. target_ongoing_requests=2 is far below the concurrency a well-tuned vLLM replica can actually hold, as chapter 6 spent an entire section computing. Using "auto" unmodified on a vLLM deployment tends to over-provision: it scales up readily toward a ceiling that was never sized for your hardware, and each of those 100 hypothetical replicas is a GPU node you're paying for.
[+] When "auto" is the right call anyway
For a lightweight, stateless service in front of your GPU fleet — the router actor's HTTP-facing sibling, an embedding microservice, anything CPU-only where a replica costs little — the defaults inside "auto" are genuinely reasonable, and writing out the full config by hand adds no value. Reach for "auto" there without hesitation, and always override it explicitly the moment a deployment holds a GPU.
7.17 Resizing a caller pool live
The endpoint URL never changes, so there is no routing table to update. What changes is the number of GPUs behind it — and the caller-side limit that was correct for two replicas is badly wrong for eight.
Recall the gap flagged back in 7.8: an ActorPool is fixed-size until something touches it, and 7.4's governor holds a max_in_flight that was set once at startup. Neither notices when the fleet resizes. Ray Serve's autoscaler adds and removes replicas; nothing in that path tells your code that the amount of capacity it should be driving just changed.
[def] The pattern: poll capacity, then resize
A small actor periodically calls the capacity API — the endpoint that reports how many replicas or GPUs are currently serving — converts that count into a concurrency limit, and pushes the new limit into the governor. The URL never enters the conversation. This is ordinary state management: the governor from 7.4 already holds max_in_flight; this just keeps that number current instead of frozen at its startup value.
# Concurrency each replica can absorb before latency degrades. From chapter 6's
# KV-cache arithmetic (6.5) - measured, not guessed.
PER_REPLICA_CONCURRENCY = 10
@ray.remote
def capacity_sync_loop(gateway, pool_manager, poll_seconds: float = 10.0):
last_seen = None
while True:
# The capacity API - NOT service discovery. One number, not a list of hosts.
replicas = get_vllm_replica_count() # e.g. {"ready": 8}
if replicas != last_seen:
limit = replicas * PER_REPLICA_CONCURRENCY
ray.get(gateway.set_max_in_flight.remote(limit))
ray.get(pool_manager.resize.remote(replicas))
last_seen = replicas
time.sleep(poll_seconds)[+] One number in, two dials out
The replica count drives both caller-side dials at once. It sets the governor's in-flight ceiling, so you saturate eight replicas without drowning two. And it sizes the embedding ActorPool from 7.8, because if vLLM capacity quadruples then the traffic reaching every other stage is about to quadruple too — scale that pool ahead of the wave rather than waiting for its own metrics to notice it has become the bottleneck. Using the fleet size as a leading indicator is the whole reason a single poll is worth this much.
[!] Scale the limit down before the GPUs go, and up after they arrive
Ordering matters, and it is asymmetric for the same reason 7.15's delays are. On the way up, a replica that has joined the fleet may still be loading weights and capturing CUDA graphs, so raising your limit the instant the count changes pushes traffic at something not yet ready — prefer a count of ready replicas over a count of existing ones. On the way down, lowering your limit promptly is exactly right: it stops you sending work toward capacity that is draining.
[retail] The capacity API is not always a separate service
It might be a Serve status endpoint, a Kubernetes API returning ready pod counts for the deployment, or a cloud provider query. The shape is what matters, not the source: a number you can poll cheaply, describing how much capacity is live right now. If nothing like it exists, the fallback is inferring capacity from observed latency — workable, but strictly worse, because you only learn you overshot after requests have already slowed down for real users.
7.18 The full elastic pipeline
Put every piece from this chapter together against a concrete timeline, because seeing it happen in sequence is what makes the two-autoscaler idea from 7.12 click.
Traffic to your RAG assistant rises through the morning. Here is what actually happens, layer by layer, minute by minute — no single component does all of this; each does exactly the one job this chapter assigned it.
- t = 0:00 — Requests start queueing on 2 vLLM replicas Ongoing requests per replica rises past target_ongoing_requests. Serve's replica autoscaler (7.14) starts its clock.
- t = 0:30 — upscale_delay_s elapses; Serve decides to add replicas It asks for the GPUs 4 more TP=4 replicas need. If the cluster already has spare capacity, replicas start immediately. If not:
- t = 0:30–2:00 — the cluster autoscaler (7.13) provisions nodes New GPU nodes join the Ray cluster. This step does not exist in every deployment — a cluster with headroom already skips straight to the next step.
- t ≈ 2:00 — new vLLM replicas finish loading Process start, weight loading, CUDA graph capture — the tens-of-seconds cost chapter 6 measured, now multiplied by however long node provisioning took.
- t ≈ 2:00 — the capacity sync loop (7.17) polls and sees 6, not 2 Within one poll interval it multiplies the new replica count by the measured per-replica concurrency and pushes a higher max_in_flight into the gateway actor. The endpoint URL never changed; only the ceiling did.
- t ≈ 2:00 — the embedding ActorPool grows in proportion Sized off the same replica count, so the caller layer scales with the layer generating its traffic rather than lagging a full detection-and-react cycle behind it.
- Traffic falls in the afternoon Ongoing requests per replica drops. Serve's downscale_delay_s (600s by default) must elapse with load continuously low before anything is removed — the asymmetry from 7.15 doing its job.
- A replica is marked for removal Serve stops routing new requests to it via the load balancer and waits for in-flight ones to drain. The capacity API reflects the lower ready-count on its next poll, and the sync loop lowers max_in_flight to match — which is why 7.17 insisted this direction should happen promptly, not cautiously.
[+] What made this possible, in one sentence
Every layer in this timeline only had to solve the one problem it owns: Serve decides how many replicas, the cluster autoscaler decides where they run, and the capacity sync loop keeps the caller-side actors from this chapter's Parts A–C consistent with whatever the other two decided. None of those three needed to know how the other two work internally — only the shape of what they hand off, which is exactly the separation of concerns this chapter built toward one scenario at a time.
[!] The one thing to load-test before trusting this in production
The gap between step 1 and step 5 — from "requests start queueing" to "new capacity is actually serving traffic" — is your real scale-up latency, and for a GPU replica it is minutes, not seconds. If your traffic spikes faster than that gap, users queue behind capacity that hasn't arrived yet regardless of how well every individual piece here is tuned. The fix isn't a smaller delay; it's min_replicas set high enough to absorb your actual spike shape, with autoscaling handling only above that floor.
7.19 Key takeaways
The eleven things worth remembering
- Tasks are for stateless work; actors are for anything that must be remembered between calls. That single question decides which primitive to reach for before any other detail matters.
- ray.get() collapses concurrency; submit everything first, collect once. Calling it right after every .remote() reproduces a sequential loop with extra steps.
- ray.wait() lets you act on whichever result arrives first, which matters specifically when some of a fan-out's dependencies are optional and tail latency is the thing you're protecting.
- Named actors with get_if_exists=True are what makes an actor into infrastructure rather than a value one process happens to hold — found from anywhere, safe against duplicate creation on restart.
- lifetime="detached" and max_restarts must be set deliberately for anything meant to outlive the process that created it. A restart brings the process back, not its prior state.
- Async actors give an I/O-bound actor real concurrency without adding processes, but max_concurrency needs the same deliberate sizing as any connection pool.
- An ActorPool is fixed-size until something resizes it — the exact gap that elastic inference has to close explicitly.
- Placement group strategy should match the actual guarantee you need: STRICT_PACK for TP shards that need NVLink, SPREAD for replicas whose fault-tolerance depends on separate nodes.
- There are two autoscalers, not one. The cluster autoscaler provisions nodes; the Serve replica autoscaler decides how many replicas to run on them. Confusing "no replica" with "no node" sends you debugging the wrong layer.
- Serve's asymmetric delays — fast up, slow down — exist to prevent flapping, because a GPU replica's startup cost (weight loading, CUDA graph capture) is expensive enough that reacting to noise is a real cost, not a theoretical one.
- The caller layer must be kept in sync with the GPU fleet explicitly. Nothing does this automatically; a small poll-and-resize loop is what connects a fleet capacity API to the gateway's in-flight ceiling and the ActorPool's size.
[def] The one-sentence version
Choose tasks or actors based on whether something needs remembering, name and detach the actors that must act as infrastructure, size their concurrency and placement deliberately, and remember that the fleet you're calling and the code calling it scale on different signals — so something has to explicitly keep the second honest about the first.
7.20 Interview drills
Ray questions test whether you reach for the right primitive because you understand the problem, or because you memorised a decorator. Every answer below starts from the mechanism.
1. When would you use a Ray actor instead of a task?
When something must be remembered across calls that a stateless function can't carry. A task runs fresh each time with no guarantee it's even the same process; an actor is a class given its own long-lived process, so its instance state accumulates exactly like an ordinary running program. A router tracking which of several backend replicas is least loaded needs an actor, because the whole point is remembering load across thousands of separate routing decisions.
I wouldn't default to actors everywhere, though. Wrapping stateless work in an actor adds process overhead and a serialisation point for no benefit, since Ray already parallelises independent tasks for free. The question I ask first is whether the work needs memory at all; only if the answer is yes does an actor enter the conversation.
2. Explain named actors and why get_if_exists matters in practice.
A named actor is created with a name and a namespace and can be retrieved from anywhere in the cluster with ray.get_actor(), without ever passing its handle around. That turns an actor from a value one process happens to hold into something closer to a small service other parts of the system can find independently.
get_if_exists matters because a real startup path isn't single-threaded across the whole system: multiple processes can start at nearly the same time and all try to create the same named actor. Without it, whichever loses the race raises an error instead of joining the winner. I'd treat get_if_exists=True as the default for any actor that might be created from more than one place, which in practice is most infrastructure actors.
3. What does lifetime="detached" actually do, and what doesn't it do?
By default an actor is tied to the process that created it; when that process exits, Ray tears the actor down too. lifetime="detached" removes that tie, so the actor keeps running independently until something explicitly kills it. That's what a named router or any piece of long-lived infrastructure needs, since it shouldn't disappear just because the script that happened to start it exited.
What it doesn't give you is resilience to crashes — that's a separate setting, max_restarts. And even with max_restarts set, a restart brings the process back and re-runs __init__; it does not restore whatever state the instance held before it crashed. If that state has to survive, it needs to be persisted outside the actor and reloaded on init. I'd be explicit about which of the two problems, lifetime or restart, I'm actually solving, because they're independent settings.
4. Your actor makes network calls to a downstream service. How do you get real concurrency out of it?
By default an actor processes one method call at a time, which is fine when the work is nearly instant but a real bottleneck when methods spend most of their time awaiting a network response. Declaring the actor's methods async def is enough for Ray to detect it automatically and run them on an internal event loop instead, so many calls can be concurrently awaiting I/O inside that one process.
The dial that actually matters after that is max_concurrency — the cap on how many calls can be in flight inside the actor at once. I'd size it the same way I'd size a connection pool: against the concurrency limit of whatever is on the other end of the call, not arbitrarily high. And I'd remember this only helps waiting; if a method does real computation rather than awaiting I/O, it still blocks every other in-flight call on that actor's single event-loop thread while it runs.
5. Why would a placement group request fail to schedule even when the cluster has enough total free GPUs?
Because STRICT_PACK asks for every bundle on one node, not just enough capacity somewhere in the cluster. Four GPUs scattered across four different nodes with one free GPU each doesn't satisfy a request for four GPUs strictly on a single node, and the placement group correctly fails rather than silently spreading across nodes.
That's usually the right failure. For a tensor-parallel group that needs NVLink between shards, spreading across nodes wouldn't be a smaller version of the same thing — it would be a correct but far slower deployment, discovered only once it was already running. I'd rather the scheduling request fail loudly at startup than silently degrade, which is exactly the trade-off STRICT_PACK is designed to make.
6. Distinguish the Ray cluster autoscaler from the Ray Serve replica autoscaler.
They operate at different layers and neither knows about the other's internals. The cluster autoscaler watches whether pending tasks and actors can be scheduled on the nodes that currently exist; when demand exceeds what's available it asks the underlying infrastructure for more nodes, and releases idle ones later. It has no concept of a "replica" or a request — it would behave the same whether the workload was language models or anything else.
The Serve replica autoscaler sits above that. It watches how loaded a deployment's replicas are against a target and decides how many replicas should exist, then relies on the cluster autoscaler underneath to actually provision the nodes for them if none are free. If I see replicas stuck pending, I'd check which layer is actually stalled — Serve deciding not to scale, or the cluster autoscaler unable to get the nodes it asked for — because the fix is completely different depending on which one it is.
What is being tested: whether you've actually deployed something with Ray Serve or are describing it secondhand.
7. Why does Ray Serve scale up in 30 seconds by default but wait 10 minutes to scale down?
Because the two mistakes cost differently. Scaling up too slowly means real users queue behind capacity that hasn't arrived, which is a direct user-facing cost, so the up-delay is kept short. Scaling down too eagerly risks flapping: a brief traffic lull triggers a removal, traffic resumes seconds later, and Serve pays a GPU replica's full startup cost — process launch, weight loading, CUDA graph capture, easily tens of seconds — to replace something it didn't need to remove in the first place.
The asymmetric delay makes that mistake structurally unlikely: only a sustained drop lasting the full ten minutes triggers a scale-down, by which point it's clearly not noise. I'd tune both delays around the actual shape of my traffic and the actual cost of a replica's cold start, rather than treating the defaults as universal — a smooth, well-understood diurnal pattern can usually tolerate a shorter downscale delay than a bursty one.
8. Your vLLM fleet scales from 2 replicas to 8 under load, behind the same stable endpoint. What has to happen for callers to actually benefit from that?
Scaling the fleet is necessary but not sufficient. Ray Serve's autoscaler decides replica count and the cluster autoscaler provisions the nodes, but nothing in that path updates the caller-side concurrency limit or resizes a downstream ActorPool doing embedding calls — those are caller-side state that Serve doesn't know exist, let alone update. The endpoint URL doesn't change, which is exactly why this is easy to miss: nothing looks broken, callers are just quietly under-driving eight replicas' worth of hardware.
I'd run a small poll-and-resize loop against a capacity API that reports how many replicas are currently ready, convert that count into a concurrency ceiling using a measured per-replica number, and push it into the gateway actor holding the in-flight count. The same signal — replica count roughly quadrupling — is also the leading indicator to grow the embedding pool ahead of the traffic increase reaching it, rather than waiting for the embedding layer's own metrics to notice it's become the bottleneck after the fact. Without that explicit sync, the eight replicas exist but the caller is still behaving as if there were two.
Where this leaves you
You can now build the layer of code that sits between a request and the GPU fleet from chapter 6: choosing tasks versus actors based on whether something needs remembering, naming and detaching the actors that act as infrastructure, sizing their concurrency deliberately, and placing them where correctness or fault tolerance actually depends on it. More importantly, you can reason about why a fleet of GPU replicas and the code calling it need to be kept in sync explicitly — they scale on different signals, at different speeds, and nothing does that reconciliation for you by default.
The thread through this chapter is the same one from 7.1: almost everything here exists because the calling code is I/O-bound and the GPU fleet it calls is not fixed in size. Tasks and async actors get concurrency out of waiting. Named, detached actors turn that concurrency into something the rest of the system can rely on. Two separate autoscalers, kept honest by a small sync loop, let the whole thing grow and shrink without a human editing a config file every time traffic changes.
Chapter 8 moves from orchestrating existing engines to giving a model the ability to act on its own: tool calling, the agent loop, and what changes when the thing making decisions about which GPU replica to call is the model itself rather than a router you wrote.