Chapter 14 · Platform and delivery
Platform and Delivery
Everything built so far has to run somewhere other than a laptop. This chapter covers containers, orchestration and pipelines through the lens of what makes AI workloads genuinely different: models measured in gigabytes, startup measured in minutes, and hardware expensive enough that idle capacity is a line item.
[!] What this chapter assumes, and what it deliberately skips
It assumes chapter 6's vLLM server and chapter 9's FastAPI service as the things being deployed, and chapter 13's branching workflow as what triggers a pipeline. Cloud examples are GCP, since that's where the catalog in chapter 12 already lives — AWS and Azure equivalents are named where the mapping is one-to-one, but not explored, because a tour of three providers teaches less than one worked properly.
There is no Helm chart authoring, no service mesh, and no Terraform. All three are real, and all three are a second chapter rather than a rushed section in this one.
14.1 The problem containers actually solve
"It works on my machine" is the joke version. The real problem is that a Python service depends on far more than its Python packages, and almost none of that is captured by a requirements file.
[def] A container image packages the filesystem an application needs, not a whole machine
The image contains the OS libraries, the Python interpreter, the installed packages and your code, all pinned together. Containers running from it share the host's kernel rather than booting their own, which is why a container starts in milliseconds where a virtual machine takes a minute. The trade is that a container cannot run a different kernel from its host — relevant mainly when GPU drivers enter the picture in 13.4.
[retail] For AI work, the dependency problem is worse than average
A vLLM service from chapter 6 depends on a specific CUDA runtime, a matching PyTorch build, and a driver on the host new enough to support both. Those constraints are genuinely hard to satisfy by hand on a fresh machine, and impossible to satisfy identically across twenty machines. This is the case where containers stop being convenient and start being the only sane option.
14.2 Images, layers, and why builds are slow
Every instruction in a Dockerfile creates a layer, layers are cached, and the cache is invalidated top-down. Understanding that one rule is most of what makes builds fast.
[def] Cache invalidation cascades downward
Docker reuses a cached layer only if that instruction and every instruction above it are unchanged. Change one line of application code, and every layer after the COPY that brought it in gets rebuilt. This is why instruction order matters enormously: things that rarely change belong near the top, and the code you edit constantly belongs at the bottom.
Reinstalls every dependency on every code change
FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["uvicorn", "main:app"]Dependencies cached until requirements change
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app"][+] Multi-stage builds keep build tools out of the shipped image
A build stage can install compilers and headers to build a wheel, and a final stage can copy just the built artifact into a clean base image. The result ships without the toolchain, which means a smaller image, a faster pull onto every node, and a smaller attack surface — three benefits from one structural change.
14.3 A Dockerfile for a Python inference service
Concrete rather than abstract: this is a reasonable starting point for the FastAPI service from chapter 9, annotated with the reasoning behind each choice.
FROM python:3.12-slim AS base
# Don't buffer stdout - otherwise logs vanish when a container is killed
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_NO_CACHE_DIR=1
WORKDIR /app
# Dependencies first, so this layer caches across code changes
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Run as a non-root user: a container escape shouldn't land on root
RUN useradd --create-home --uid 10001 appuser
USER appuser
COPY --chown=appuser:appuser . .
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"][!] Three mistakes that show up constantly
- Binding to 127.0.0.1
- Inside a container that means "only reachable from inside this container." It must be 0.0.0.0 to accept traffic from outside, and this is a genuinely common half-hour of confusion.
- Running as root
- The default if you don't say otherwise. Most base images let you add a user in two lines, and many clusters refuse to run root containers at all.
- Using the latest tag
- It makes builds unreproducible and rollbacks ambiguous. Pin a real version, and tag your own images with something traceable like the Git commit SHA (14.19).
[+] A .dockerignore is the cheapest build speedup available
Without one, COPY . . sends your entire working directory to the builder — .git, virtual environments, node_modules, model checkpoints, test fixtures. Excluding those routinely cuts build context from gigabytes to megabytes, and prevents the genuinely bad outcome of a stray .env file being baked into a shipped image.
14.4 GPU containers break the usual assumptions
Almost everything above applies unchanged to a GPU workload, except for the parts that matter most: the image is enormous, the driver is on the host, and the hardware cannot be shared casually.
[def] The driver stays on the host; the toolkit ships in the image
Containers share the host kernel, and the NVIDIA driver is a kernel component — so it must be installed on the host, not in your image. What ships inside the image is the CUDA runtime and libraries your framework links against. The container runtime then exposes the host's GPU devices into the container. This split is why a CUDA image that works on one machine fails on another whose host driver is too old.
| Ordinary API service | GPU inference service | |
|---|---|---|
| Image size | Tens to low hundreds of MB | Several GB before model weights are added |
| Base image | python:3.12-slim or similar | A CUDA runtime image matching the framework build |
| Host requirement | A container runtime | Runtime, plus a sufficiently new NVIDIA driver |
| Startup time | Seconds | Minutes, dominated by loading weights into VRAM |
| Sharing | Many replicas per node freely | Typically one process per GPU — VRAM isn't overcommittable |
[!] Never bake model weights into the image
It's tempting, and it makes the image reproducible. It also produces a fifty-gigabyte artifact that has to be pushed to a registry, pulled onto every node, and rebuilt in full every time the model changes independently of the code. Weights belong in object storage, fetched at startup or mounted from a shared volume — 14.12 covers the options and their trade-offs directly.
14.5 Compose: the right tool more often than admitted
Kubernetes is the default answer to "how do we run containers in production," and for a great many systems it is more machinery than the problem requires.
[def] Compose runs several containers together on one machine
One YAML file declares the services, the network between them, and their volumes, and a single command brings all of it up. There is no cluster, no scheduler, and no control plane — which is exactly the point. Everything runs on one host, so if that host dies, everything is down.
[+] When Compose is genuinely the correct choice
Local development, where a developer needs the API, Postgres and Redis from chapter 12 running together in one command. CI, where a test suite needs real dependencies rather than mocks. Single-machine deployments where an hour of downtime during a host failure is acceptable. A batch job like chapter 12's inference pipeline, which runs on one big VM and doesn't need orchestrating at all. Reaching for Kubernetes in any of these adds a cluster to operate and solves nothing that was actually broken.
[!] The honest limits
No scheduling across machines, no automatic rescheduling when a node fails, no rolling deployment with health gating, and no horizontal scaling beyond one host's capacity. The moment you need any of those — and "the service must survive a machine failure" is the usual first one — you've outgrown it, and 14.6 is the answer.
14.6 What Kubernetes is actually for
Kubernetes is often described as a container orchestrator, which is accurate and unhelpful. It is more usefully described as a control loop that continuously works to make reality match a description you wrote.
[def] Declarative desired state, reconciled continuously
You declare "three replicas of this image, each needing one GPU and 40GB of memory." Kubernetes places them on nodes with capacity, and then keeps checking. A pod crashes, it restarts it. A node dies, it reschedules that node's pods elsewhere. You never issue the instruction to recover — the gap between declared and actual state is what triggers action, permanently and automatically.
[retail] The specific thing this buys an AI platform
Chapter 7's Ray fleet and chapter 6's vLLM replicas both assume something is keeping a given number of GPU workers alive across a pool of machines, replacing failures without a human paging. That is precisely the job Kubernetes does. It is also what makes GPU capacity poolable — nodes are a shared resource that scheduled work draws from, rather than named machines somebody has to assign by hand.
[!] The cost is real and worth naming
A cluster is infrastructure that itself needs upgrading, monitoring, securing and debugging, plus a genuinely large vocabulary before anyone can be productive. On a managed service like GKE the control plane is somebody else's problem, which removes the hardest part — but a team of three shipping one service still usually gets further with a simpler platform, and 14.17 makes that comparison concrete.
14.7 The eight objects that cover most needs
Kubernetes has dozens of object types. These eight cover the overwhelming majority of what an application team actually writes.
| Object | What it is | When you write one |
|---|---|---|
| Pod | One or more containers scheduled together, sharing a network address | Rarely directly — something else usually creates them |
| Deployment | Keeps N identical, interchangeable pods running and handles rollouts | For any stateless service, including an inference server |
| Service | A stable internal address and load balancing across a set of pods | Whenever anything else needs to reach your pods |
| Ingress | Routes external HTTP traffic to Services, usually terminating TLS | To expose something outside the cluster |
| ConfigMap | Non-secret configuration, injected as env vars or files | Model name, batch size, feature flags |
| Secret | The same, for credentials — see 14.16 on its real guarantees | API keys, database passwords |
| Job / CronJob | Runs to completion, once or on a schedule, rather than forever | Batch inference, nightly exports, migrations |
| StatefulSet | Like a Deployment, but pods get stable identities and their own storage | Databases and similar — rarely for inference |
[+] Deployment for inference, StatefulSet almost never
A model server holds no unique per-replica state: any replica can serve any request, and replacing one loses nothing. That is exactly the Deployment case. StatefulSets exist for workloads where replica three is meaningfully different from replica one and must keep its own disk — a database, a queue broker. Using one for inference adds ordering constraints during rollouts that make deployments slower for no benefit.
14.8 Requests, limits, and the OOMKill
Two numbers per container that decide where it gets scheduled and when it gets killed. Getting them wrong is behind a large share of mysterious production restarts.
[def] Requests are for scheduling; limits are for enforcement
- Request
- What the pod is guaranteed. The scheduler uses this to decide which node has room, and it reserves that capacity whether or not the pod uses it.
- Limit
- The ceiling. Exceed the memory limit and the container is killed immediately — the infamous OOMKilled status. Exceed the CPU limit and it is throttled instead, which is slow rather than fatal.
[!] Memory and CPU behave completely differently when exceeded
This asymmetry surprises people repeatedly. CPU is compressible: exceeding the limit slows you down. Memory is not: exceeding it kills the container instantly, mid-request, with no graceful shutdown. A pod that restarts every few hours with OOMKilled and no application error in its logs is almost always a memory limit set below what the workload genuinely needs at peak.
[retail] For inference pods, set request equal to limit
A model server's memory footprint is dominated by weights and KV cache, which are large, predictable, and allocated up front — there's no bursty tail to accommodate. Setting request and limit to the same value gives the pod a guaranteed allocation the scheduler fully accounts for, and removes the failure mode where a node's pods collectively overcommit and one gets killed because a neighbour grew. Chapter 6's memory budgeting is exactly the arithmetic that produces this number.
14.9 Scheduling GPUs, and why pods sit Pending
GPUs are requested like memory and CPU, but they behave differently in one important way: they cannot be oversubscribed, so a pod either gets a whole device or waits.
resources:
requests:
memory: "80Gi"
cpu: "8"
nvidia.com/gpu: 1 # whole devices only - no fractions
limits:
memory: "80Gi"
cpu: "8"
nvidia.com/gpu: 1 # must equal the request[!] A GPU pod stuck in Pending: the four usual causes
- No node has a free GPU
- The most common. Every GPU in the pool is already allocated to another pod, and unlike CPU there's no oversubscription to fall back on.
- The device plugin isn't running
- Nodes only advertise nvidia.com/gpu capacity if the NVIDIA device plugin is installed and healthy. Without it, the cluster believes there are no GPUs at all.
- Taints without matching tolerations
- GPU node pools are usually tainted to keep ordinary workloads off expensive hardware. Your pod needs the matching toleration or the scheduler won't place it there.
- Other resources don't fit
- A GPU is free but the node lacks the 80Gi of RAM you also requested. The GPU gets blamed; the memory request is the real constraint.
[+] Diagnose with describe, not logs
A Pending pod has no logs because nothing has started. kubectl describe pod <name> shows scheduler events at the bottom, which state plainly why placement failed — insufficient nvidia.com/gpu, or an untolerated taint. That output is the answer nearly every time, and it's the first thing to reach for rather than the last.
14.10 Probes when a model takes minutes to load
Kubernetes health checks assume services start in seconds. A model server loading tens of gigabytes of weights does not, and the default probe configuration will kill it repeatedly in a loop that looks like a crash.
[def] Three probes, three different questions
- Startup probe
- "Has it finished starting yet?" While this is failing, the other two are suspended entirely. This is the one that saves slow-starting model servers.
- Readiness probe
- "Should traffic be sent here right now?" Failing removes the pod from the Service's endpoints without killing it — correct for a replica that's temporarily overloaded.
- Liveness probe
- "Is this process wedged and beyond recovery?" Failing restarts the container. Genuinely dangerous to misconfigure.
startupProbe:
httpGet: { path: /health, port: 8000 }
periodSeconds: 10
failureThreshold: 60 # allows up to 10 minutes to start
readinessProbe:
httpGet: { path: /health, port: 8000 }
periodSeconds: 5 # pulled from load balancing quickly if unhealthy
livenessProbe:
httpGet: { path: /health, port: 8000 }
periodSeconds: 30
failureThreshold: 3 # deliberately slow to conclude "restart it"[!] An aggressive liveness probe turns a slow model into a restart loop
Without a startup probe, the liveness probe begins checking immediately, fails while weights are still loading, and restarts the container — which starts loading from scratch again, and repeats forever. The pod shows CrashLoopBackOff and nothing in the application logs explains it, because the application never did anything wrong. A startup probe with a generous failureThreshold is the fix, and it should be the default for any model server.
[retail] Liveness should test the process, not its dependencies
If the liveness endpoint checks that the vector database from chapter 4 is reachable, then a blip in that database restarts every inference pod simultaneously — turning a partial outage into a total one, and throwing away several minutes of model loading in the process. Liveness answers "is this process wedged." Dependency health belongs in readiness at most, and often just in monitoring.
14.11 Autoscaling on the wrong metric
The default autoscaler scales on CPU utilisation, which is very close to meaningless for a GPU inference service.
[!] CPU is idle while the GPU is saturated
During generation, the CPU mostly waits on the GPU. A vLLM replica can be completely saturated — queue backing up, latency climbing — while reporting low CPU utilisation. An autoscaler watching CPU concludes everything is fine and never adds capacity. Meanwhile a burst of short requests can spike CPU without the GPU being busy at all, and it scales up for no reason.
| Metric | Why it reflects real load |
|---|---|
| Requests waiting in the queue | Directly measures demand exceeding current capacity — usually the best single signal |
| Time to first token | The user-visible symptom of saturation, and what an SLO is usually written against |
| KV cache utilisation | vLLM's actual constraint (chapter 6); near-full cache means throughput is about to degrade |
| GPU utilisation | Better than CPU, though a busy GPU isn't automatically an overloaded one |
[retail] Scaling GPU replicas is slow, so scale earlier than feels necessary
Adding a stateless API pod takes seconds. Adding a GPU replica may mean provisioning a node, pulling a multi-gigabyte image, and loading weights — several minutes before it serves a single request. An autoscaler tuned to react at the moment of saturation is therefore always minutes late. Scale on a leading indicator, keep enough headroom to cover the provisioning window, and accept some idle capacity as the price of not dropping traffic during a spike.
14.12 Getting model weights into a container
14.4 ruled out baking weights into the image. That leaves three real options, and the choice measurably affects how quickly a replica can start serving.
| Approach | Startup cost | Best when |
|---|---|---|
| Download from object storage at startup | Full download every time a pod starts | Simple, and fine when replicas are long-lived |
| Shared read-only network volume | No download; slower first reads, then cached | Many replicas share one model, on one cloud's storage |
| Pre-populated local disk on the node | Fastest by far — already on local SSD | Fixed node pool, model changes rarely |
[+] An init container keeps the download out of your application code
An init container runs to completion before the main container starts, so a small image whose only job is fetching weights into a shared volume keeps that concern entirely separate from the inference server. It also fails cleanly and visibly: a failed download shows as the init container failing, rather than as a confusing application error several minutes into startup.
[!] The weights are a versioned artifact, exactly like the image
Pointing at gs://models/current/ means a rollback of the code doesn't roll back the model, and two replicas started an hour apart can be serving different weights without anything indicating it. Version the path explicitly, pin it in configuration alongside the image tag, and treat "which model version is this replica running" as something the deployment declares rather than something the storage bucket decides.
14.13 What a pipeline should actually do
A CI/CD pipeline's job is to make the path from a merged pull request to running code boring, repeatable, and impossible to do halfway.
[def] CI and CD answer different questions
- Continuous integration
- Runs on every push and every pull request: lint, type check, unit tests, build the image. Answers "is this change safe to merge?" — and it's the thing that should be green before anyone is asked to review (13.17).
- Continuous delivery / deployment
- Runs after merge: push the image to a registry, roll it out, verify it. Answers "is this change now running?" Delivery stops at a manual approval; deployment goes all the way automatically.
On pull request:
lint + type check # fast, fails in seconds
unit tests # no network, no GPU
build image # proves the Dockerfile still works
integration tests # Compose: API + Postgres + Redis (13.5)
On merge to main:
build + tag image # tagged with the commit SHA (13.19)
push to Artifact Registry
deploy to staging # automatic
smoke test staging # a few real requests through the real path
deploy to production # gated: manual approval or automatic
verify + watch # 13.15's rollback trigger lives here[+] Order stages fastest-first, so failure is cheap
A lint error should fail in ten seconds, not after a six-minute image build. Ordering stages by increasing cost means the common, trivial mistakes are reported almost immediately, and the expensive stages only run against changes that have already cleared the cheap checks. This is one of the few pipeline decisions that measurably changes how it feels to work in a repository day to day.
[!] Don't put GPU tests in the pull-request path
GPU runners are expensive and usually scarce, so putting a full model-loading test on every push creates a queue that blocks the whole team. Keep the PR path CPU-only, mocking the inference call, and run the genuine GPU verification on merge to main or on a schedule. The point is fast feedback where changes are frequent, and thorough verification where the cost is justified.
14.14 Rolling, blue-green, and canary
Three ways to replace running code with new code, differing in how much traffic is exposed to a bad version before anyone notices.
| Strategy | How it works | Extra capacity needed |
|---|---|---|
| Rolling | Replace pods a few at a time, waiting for each batch to become ready | One extra pod's worth |
| Blue-green | Stand up a complete second environment, switch all traffic at once, keep the old one warm | Double, for the duration |
| Canary | Send a small percentage of traffic to the new version, watch, then increase | A few extra pods |
[retail] Blue-green is usually unaffordable for GPU services
Doubling a fleet of stateless API pods for ten minutes is a rounding error. Doubling a fleet of GPU nodes means provisioning an entire second set of the most expensive hardware in the system, possibly hitting a quota, for a deployment that takes several minutes per replica to become ready. Rolling with a small surge, or canary, is the realistic choice for model serving — and this is a good example of a general platform practice that needs adjusting specifically because the workload is GPU-bound.
[+] Canary is worth the complexity when the failure is subtle
A crash is caught by any strategy, because readiness probes stop the rollout. What canary catches that rolling doesn't is a version that starts fine and is quietly worse — higher latency, degraded output quality after a model change, an error rate that's up but not catastrophic. Routing five percent of traffic and comparing metrics against the other ninety-five is the only way to see that before everyone is affected.
14.15 Rollback: the plan you need before you deploy
The question that matters is not whether a deployment will eventually go wrong, but how many minutes it takes to get back to the last good state when it does.
[def] Rollback is a first-class operation, not an improvisation
Kubernetes keeps previous Deployment revisions, so kubectl rollout undo deployment/<name> returns to the prior image immediately. That works because every deployment is an immutable, tagged image (14.19) rather than a mutable tag that has since been overwritten — which is precisely why the latest tag makes rollback ambiguous.
[!] Database migrations are what make rollback hard
Code rolls back in seconds; a schema change does not. If version two dropped a column version one still reads, rolling back the code leaves the old version querying a column that no longer exists. The standard discipline is expand-then-contract: deploy the schema change so both versions work (add the new column, keep the old), ship the code, and only remove the old column in a later release once no running version needs it. It's two deployments instead of one, and it's what makes the first one safely reversible.
[+] Decide the rollback trigger before deploying, not during the incident
"Error rate above two percent for five minutes, or p99 latency doubled" is a decision that's easy to make calmly beforehand and very hard to make well at eleven at night with a dashboard going red. Writing it down — ideally automating it — converts a judgement call under pressure into a threshold that either was or wasn't crossed.
14.16 Secrets, and where they must not live
Chapter 13 covered keeping credentials out of Git. The deployment side has its own version of the same problem, plus one widely misunderstood detail about Kubernetes Secrets.
[!] A Kubernetes Secret is base64-encoded, not encrypted
Base64 is an encoding, not a cipher — anyone who can read the Secret object can trivially decode it. Secrets are better than a plain ConfigMap because access is controlled separately and they're kept out of most logs, but the protection comes from the access controls around them, not from the encoding. Treating "it's in a Secret" as equivalent to "it's encrypted" is a genuinely common and consequential misunderstanding.
| Location | Verdict |
|---|---|
| Hardcoded in source | Never. It's in history permanently, and compromised the moment the repo is cloned. |
| Baked into a container image | Never. Anyone who can pull the image can extract it. |
| Kubernetes Secret | Acceptable, with tight access control and encryption at rest enabled. |
| A managed secret manager | Better — central rotation, audit logging, and no long-lived copy in the cluster. |
| Workload identity, no credential at all | Best. The pod's own identity authorises it, so there's no secret to leak. |
[+] The strongest version is having no credential to steal
On GCP, Workload Identity lets a Kubernetes service account act as a Google service account, so a pod reads from Cloud Storage or BigQuery using its own identity rather than a key file mounted from a Secret. AWS and Azure have direct equivalents. This removes the entire category of "a long-lived key leaked" — which is a meaningfully better outcome than protecting that key carefully, and it's the default worth reaching for on any of the three clouds.
14.17 Picking a compute service, honestly
Kubernetes is not the automatic answer. GCP offers several ways to run a container, and the cheapest one that meets the requirement is usually the right call.
| Service | Good for | The catch |
|---|---|---|
| Cloud Run | The chapter 9 API: stateless HTTP, scales to zero, no cluster to run | Request-scoped billing suits bursty traffic, less so a constantly busy service |
| Compute Engine VM | Chapter 12's batch inference job: one big machine, run it, shut it down | You own patching, monitoring, and restarts |
| GKE | A fleet of GPU model servers needing scheduling, self-healing, autoscaling | A cluster to operate, and real Kubernetes expertise required |
| Vertex AI endpoints | Serving a model without building a serving platform at all | Less control over the serving stack than running vLLM yourself |
[retail] A realistic split across this course's systems
Chapter 9's API on Cloud Run, because it's stateless HTTP that benefits from scaling to zero overnight. Chapter 6's vLLM replicas on GKE with a GPU node pool, because they need pooled expensive hardware, health-gated rollouts, and self-healing. Chapter 12's batch job on a single large Compute Engine VM, started for the run and stopped afterwards. Three different services, chosen by what each workload actually needs rather than by standardising on one.
[!] Cloud Run and a GPU model server are usually a poor fit
Scale-to-zero is exactly wrong for a service whose cold start means pulling a multi-gigabyte image and loading weights for several minutes. The first request after an idle period pays all of that, and request-scoped billing doesn't help when the model must stay resident anyway. Keep the serverless option for the stateless API in front, and let the GPU tier be something that stays warm.
14.18 GKE specifics worth knowing
Four GKE-specific details that come up immediately when running GPU workloads, and are easier to know in advance than to discover during an incident.
[def] The four
- Separate GPU node pools
- Create a dedicated pool for GPU nodes, tainted so ordinary pods can't land on them. Otherwise a stateless API replica occupies a node you're paying GPU rates for.
- Node auto-provisioning has a lead time
- Scaling a GPU pool from zero means allocating a machine, pulling the image, and installing drivers before a pod even starts. Minutes, not seconds — which is exactly why 14.11 argued for scaling on a leading indicator.
- Autopilot removes node management, and some control
- Autopilot mode manages nodes for you and bills per pod. Convenient, but it constrains what you can configure at the node level, which matters more for GPU workloads than for ordinary services.
- Workload Identity should be on from the start
- It's the mechanism behind 14.16's "no credential to steal," letting pods reach Cloud Storage and BigQuery as themselves. Retrofitting later means unpicking every mounted key file.
[+] Preemptible and Spot nodes fit batch work specifically
Spot GPU nodes cost substantially less than on-demand, with the condition that they can be reclaimed at short notice. That's unacceptable for a user-facing inference endpoint and entirely reasonable for chapter 12's overnight batch job, provided the job checkpoints its progress and can resume. The decision is simply whether an interruption costs a retry or costs a user a failed request.
14.19 Artifact Registry and image promotion
How images are tagged determines whether a rollback is a single command or an investigation, so it's worth deciding deliberately rather than by habit.
[def] Tag with the commit SHA, and promote that same image
Build once, tag with the Git commit SHA, and deploy that exact tag through staging and then production. The image tested in staging is then bit-for-bit the image running in production, and every running replica traces back to one specific commit. Rebuilding per environment breaks that guarantee: two builds of the same commit can differ if any upstream dependency moved in between.
SHA=$(git rev-parse --short HEAD)
IMAGE="us-central1-docker.pkg.dev/$PROJECT/services/inference-api:$SHA"
docker build -t "$IMAGE" .
docker push "$IMAGE"
# Deploy that exact image - never a floating tag
kubectl set image deployment/inference-api api="$IMAGE"
kubectl rollout status deployment/inference-api # waits, and fails if it stalls[+] rollout status is what makes a pipeline honest
kubectl set image returns immediately, having only recorded an intention. Without kubectl rollout status after it, a pipeline reports a green deployment while the new pods are still crash-looping. That one command is the difference between "the deployment was accepted" and "the new version is actually running and healthy," and it's what a rollback trigger (14.15) hangs off.
[!] Registries accumulate cost quietly
Every commit producing a multi-gigabyte GPU image adds up fast, and nobody notices because no single push is expensive. Set a cleanup policy that keeps tagged releases and expires untagged and old development images automatically. It's a five-minute configuration that prevents a storage bill nobody budgeted for.
14.20 The signals worth alerting on
An alert that fires without requiring action trains everyone to ignore alerts. The bar worth holding is that a page means a human must do something now.
[def] Alert on symptoms users feel, not on causes
High GPU utilisation is not a problem — it's the hardware doing the job it was bought for. Requests failing, or time to first token past what the product promised, are problems. Alerting on causes produces noise, because most causes are either harmless or self-correcting; alerting on symptoms produces pages that always correspond to something genuinely wrong.
| Signal | Why it earns a page |
|---|---|
| Error rate above the SLO threshold | Users are getting failures right now |
| p99 time to first token past target | The experience is degraded even though nothing is technically failing |
| Queue depth growing without recovering | Demand has exceeded capacity and it isn't self-correcting |
| Pods restarting repeatedly | Usually an OOMKill (14.8) or a probe misconfiguration (14.10) |
| GPU pods Pending beyond a few minutes | Capacity or quota is genuinely exhausted (14.9) |
| Spend rate above the daily budget | Something is scaling that shouldn't be, and it's expensive (14.22) |
[!] An LLM system can fail while every infrastructure metric looks perfect
Pods healthy, latency normal, error rate zero — and the model returning confident nonsense after a version change, or a truncated response because a max_tokens default moved. Infrastructure monitoring cannot see any of that. Output-quality checks belong alongside it, which is chapter 8's evaluation discipline applied continuously in production rather than once before launch.
14.21 Logs, metrics, and traces for LLM systems
The standard three pillars apply, with one addition that matters specifically because every request costs real money and takes an unusually long time.
[def] What each one is for
- Metrics
- Numbers over time, cheap to keep. Request rate, latency percentiles, queue depth, tokens per second. Good for alerting and dashboards, useless for explaining one specific bad request.
- Logs
- Discrete events with context. Structured as JSON with a request ID, not free text, so they can actually be searched when something goes wrong.
- Traces
- One request's journey across every service it touched, with timing per hop. The only practical way to answer "where did those eight seconds go" in a system with retrieval, reranking, and generation stages.
[retail] Log tokens and cost per request, from day one
Prompt tokens, completion tokens, model version, and latency, on every request, tagged by feature or tenant. This is the data that answers "why did the bill triple" and "which feature is responsible," and it is effectively impossible to reconstruct retroactively. It's a handful of extra fields in a structured log line and it pays for itself the first time someone asks where the spend went.
[!] Do not log prompts and completions carelessly
User prompts routinely contain personal data, and completions can reproduce it. Logging them wholesale creates a compliance problem in a system that was never designed to hold that data. Log identifiers, token counts, and metadata by default; log content only with explicit consent, retention limits, access controls, and a redaction step — the same care chapter 8 applied to what an agent is allowed to do with data it can reach.
14.22 GPU cost control that actually works
GPU capacity is the dominant line item in almost every AI platform, and most overspend comes from a small number of avoidable patterns.
| Pattern | Fix |
|---|---|
| GPU nodes idle overnight and at weekends | Scale the pool down on a schedule; most internal tools have no traffic at 3am |
| Development and staging GPUs running continuously | Scale to zero when unused, or share one environment rather than one per team |
| A model far larger than the task needs | Evaluate a smaller model honestly — chapter 1's argument with direct hardware cost attached |
| Batch work on on-demand hardware | Spot or preemptible nodes with checkpointing (14.18) |
| Repeated identical prompts | Cache responses; chapter 6's prefix caching for shared prompt prefixes |
| Nobody knows which team is spending what | Label every workload and break the bill down — unattributed cost never gets optimised |
[+] Utilisation is the number to watch, not total spend
A large bill for hardware running at eighty percent utilisation is a business succeeding. A smaller bill for hardware at eight percent is waste regardless of the absolute figure. Tracking utilisation alongside spend prevents both mistakes: panicking about a bill that's growing because usage is growing, and ignoring an expensive fleet that's mostly idle.
14.23 A worked deployment: vLLM behind an API
Everything in this chapter, assembled into one concrete architecture for the system this course has been building.
[!] The system being deployed
Chapter 11's React app calls chapter 9's FastAPI service, which streams tokens over SSE. That service retrieves context from the chapter 4 vector database and calls a fleet of chapter 6 vLLM replicas for generation. Chapter 12's catalog lives in BigQuery, with a nightly batch job enriching it.
| Component | Where it runs | Why |
|---|---|---|
| React app | Static hosting behind a CDN | It's compiled files; no server needed |
| FastAPI service | Cloud Run, min instances above zero | Stateless HTTP; a warm floor avoids cold starts on the streaming path |
| vLLM replicas | GKE, tainted GPU node pool | Needs pooled GPUs, health-gated rollouts, self-healing (14.6) |
| Model weights | Cloud Storage, versioned path, init container | Keeps the image small and the model independently versioned (14.12) |
| Vector database | GKE StatefulSet, or a managed service | Genuinely stateful — the one place a StatefulSet is right (14.7) |
| Nightly batch job | Compute Engine VM on Spot, started by a scheduler | Interruptible work on the cheapest hardware (14.18) |
[+] The deployment path, end to end
A pull request runs lint, tests, and a CPU-only integration suite against Compose (14.13). On merge, the image is built once and tagged with the commit SHA, pushed to Artifact Registry, and deployed to staging automatically. Smoke tests run real requests through the real path. Production is a rolling update with a small surge, gated on kubectl rollout status, with the rollback trigger agreed in advance (14.15). Weights are pinned to a version in configuration, so a code rollback doesn't silently change which model is serving.
[retail] The two failure modes this design is specifically shaped around
A GPU replica dying mid-stream, which Kubernetes replaces automatically while the API's retry logic covers the in-flight request. And a traffic spike arriving faster than GPU capacity can be provisioned, which is why the autoscaler watches queue depth rather than CPU (14.11) and why some headroom is kept deliberately idle. Both are consequences of the same underlying fact: the expensive tier is slow to add and unhelpful to overcommit.
14.24 Key takeaways
The twelve things worth remembering
- Order Dockerfile instructions by how often they change. Dependencies before code, because cache invalidation cascades downward and rebuilding every layer on every code edit is pure waiting.
- GPU containers break the normal rules. The driver lives on the host and the CUDA runtime ships in the image, images are gigabytes before weights, startup is minutes, and VRAM cannot be overcommitted the way CPU can.
- Never bake model weights into an image. Version them separately in object storage and fetch them at startup, so the model and the code can each be rolled back without the other.
- Compose is the right answer more often than people admit. Local development, CI, and single-machine batch jobs need no cluster; Kubernetes earns its complexity when work must survive a node failure and be scheduled across machines.
- Kubernetes is a reconciliation loop, not a deployment tool. You declare desired state and it continuously closes the gap, which is what makes self-healing and pooled GPU capacity possible.
- Requests schedule, limits enforce, and memory is not compressible. Exceeding a CPU limit throttles; exceeding a memory limit kills the container instantly. For inference pods, set request equal to limit.
- A model server needs a startup probe. Without one, the liveness probe kills the container mid-load and it restarts forever, showing a crash loop with nothing wrong in the application logs.
- Never autoscale a GPU service on CPU. The CPU idles while the GPU saturates. Scale on queue depth, time to first token, or KV cache utilisation, and scale early because adding a GPU replica takes minutes.
- Build the image once, tag it with the commit SHA, and promote that exact artifact. Rebuilding per environment breaks the guarantee that what you tested is what's running, and makes rollback ambiguous.
- Rollback is a plan, not a reaction. Decide the trigger before deploying, and use expand-then-contract migrations so a code rollback never hits a schema that moved out from under it.
- A Kubernetes Secret is base64-encoded, not encrypted. The protection comes from access controls around it; the strongest option is workload identity, where there's no long-lived credential to leak at all.
- Alert on symptoms users feel, and watch utilisation rather than raw spend. A big bill at high utilisation is a business working; a small bill on mostly-idle GPUs is waste.
[def] The one-sentence version
Containers make an AI service reproducible, Kubernetes makes a fleet of them self-healing, and a pipeline makes shipping boring — but every default in that stack assumes services that start in seconds and scale in seconds, so the real skill is knowing which of those defaults to change when the workload is a multi-gigabyte model on hardware you're paying for by the minute.
14.25 Interview drills
Platform questions are usually scenarios with a constraint attached. The assessment is whether you reach for the appropriate amount of machinery, and whether you know what AI workloads change about the standard answers.
1. Your GPU pod is stuck in Pending. How do you debug it?
Pending means it hasn't been scheduled, so there are no logs to read — I'd go straight to kubectl describe pod and read the scheduler events at the bottom, which usually state the reason outright.
The four common causes are: every GPU in the pool is already allocated, since GPUs can't be oversubscribed; the NVIDIA device plugin isn't running, so nodes never advertise GPU capacity at all; the GPU nodes are tainted and my pod lacks the toleration; or the GPU is free but some other request, usually memory, doesn't fit on that node. That last one is easy to misdiagnose because the GPU gets blamed for what's actually a memory constraint.
2. A model server keeps restarting with CrashLoopBackOff, but the application logs show nothing wrong. What's your first guess?
Probe configuration, specifically a missing startup probe. Model servers take minutes to load weights into VRAM, and the liveness probe starts checking almost immediately, fails while loading is still in progress, and restarts the container — which starts loading from scratch and repeats forever. The logs look clean because the application never actually did anything wrong.
The fix is a startup probe with a generous failure threshold, which suspends the liveness and readiness probes until the service reports healthy once. My second guess would be an OOMKill from a memory limit set below the real peak, which I'd confirm from the pod's last state and exit code rather than the logs.
3. Why shouldn't you autoscale an LLM inference service on CPU utilisation?
Because the CPU mostly waits on the GPU during generation. A replica can be completely saturated — queue growing, time to first token climbing — while reporting low CPU, so the autoscaler concludes everything is fine and never adds capacity. The inverse also happens: a burst of short requests spikes CPU without the GPU being busy, and it scales up pointlessly.
I'd scale on queue depth or time to first token, since those measure the thing users actually experience, or on KV cache utilisation, which is the real constraint in vLLM. And I'd scale earlier than feels necessary, because adding a GPU replica can mean provisioning a node, pulling a multi-gigabyte image, and loading weights — minutes before it serves anything, so reacting at the moment of saturation is always too late.
4. Would you use Kubernetes for a nightly inference job over a large catalog?
Probably not. That workload is one process on one big machine, reading data, calling a model, and writing results. There's nothing to schedule across nodes, nothing to keep alive indefinitely, and no traffic to load balance, which means Kubernetes would add a cluster to operate without solving anything that was broken.
I'd run it on a single VM, started on a schedule and shut down when finished, ideally on spot or preemptible hardware with checkpointing since an interruption just costs a retry rather than a failed user request. If the job later needed to fan out across many machines with retries and dependencies between stages, that's when an orchestrator earns its place — and even then a Kubernetes Job might be enough rather than a full serving setup.
5. How do you get a 40GB model into a container without a 40GB image?
Keep the weights out of the image entirely and treat them as a separately versioned artifact in object storage. The usual pattern is an init container whose only job is downloading the weights into a volume the main container mounts, which keeps that concern out of the application and makes a failed download visible as an init failure rather than a confusing startup error.
The alternatives are a shared read-only network volume, which avoids repeated downloads when many replicas use the same model, or pre-populating local SSD on a fixed node pool, which is fastest but least flexible. Whichever I picked, I'd pin the exact model version in configuration rather than pointing at something like a "current" path, because otherwise a code rollback doesn't roll back the model and two replicas can end up serving different weights.
6. Walk me through deploying a change safely, including how you'd roll it back.
The image is built once on merge and tagged with the commit SHA, then that exact image is promoted through staging to production — never rebuilt per environment, because then the thing tested isn't the thing running. Staging gets a smoke test through the real path, then production is a rolling update gated on kubectl rollout status, which is what makes the pipeline honest about whether the new pods are actually healthy rather than merely accepted.
Rollback is kubectl rollout undo, which works precisely because every deploy is an immutable tagged image. The part that needs planning is the database: I'd use expand-then-contract migrations so both versions work against the schema, and only drop the old column in a later release. And I'd agree the rollback trigger — something like error rate above two percent for five minutes — before deploying, since that's much easier to decide calmly than during an incident.
7. Is a Kubernetes Secret secure?
Not by itself, and this is a common misunderstanding. Secret values are base64-encoded, which is an encoding rather than encryption — anyone who can read the object can trivially decode it. What makes Secrets better than a ConfigMap is that access can be controlled separately and they're kept out of most logs, so the protection comes from the controls around them, not the encoding.
I'd enable encryption at rest, restrict access tightly, and prefer a managed secret manager for rotation and audit logging. The strongest option is removing the credential entirely — on GCP, Workload Identity lets a pod authenticate to Cloud Storage or BigQuery as itself, so there's no long-lived key to leak. That's a better outcome than protecting a key carefully.
8. When would you choose Cloud Run over GKE?
For stateless HTTP services where I don't want to operate a cluster — the API layer in front of a model, for instance. It scales to zero, handles TLS and routing, and removes essentially all the platform work. For a service with quiet periods, that's a genuine saving in both cost and attention.
I'd choose GKE when I need pooled GPU scheduling, health-gated rollouts across a fleet, and self-healing — which is exactly the model-serving tier. And I'd specifically avoid Cloud Run for GPU serving, because scale-to-zero is the wrong behaviour when a cold start means pulling a huge image and loading weights for several minutes. Mixing both is normal: serverless for the stateless front, a cluster for the expensive stateful tier behind it.
9. What would you monitor for an LLM service that ordinary infrastructure monitoring misses?
Output quality and cost, neither of which shows up in pod health. A system can have healthy pods, normal latency and a zero error rate while returning confident nonsense after a model version change, or truncating responses because a token limit moved. Infrastructure monitoring is structurally blind to that, so it needs continuous evaluation against known cases, not just a check before launch.
On cost, I'd log prompt tokens, completion tokens, model version and latency per request, tagged by feature or tenant. That's the data that answers "why did the bill triple" and it's effectively impossible to reconstruct after the fact. I'd also be careful not to log prompt and completion content wholesale, since that routinely contains personal data the logging system was never designed to hold.
10. Your GPU costs doubled this month. How do you find out why?
First I'd check utilisation alongside spend, because those tell very different stories. If utilisation held steady and spend doubled, usage genuinely grew and the business is working. If utilisation halved while spend doubled, I'm paying for idle hardware and something scaled up without scaling back down.
Then I'd break the bill down by label — which assumes every workload was labelled, which is why that matters from day one, since unattributed cost never gets optimised. The usual culprits are development and staging GPUs left running, production nodes never scaling down overnight, batch work on on-demand rather than spot hardware, and an autoscaler that scaled up during a spike and never released the capacity. I'd also check whether repeated identical prompts could be cached before assuming more hardware is the answer.
Where this leaves you, and the course
You now have a complete path from "what is a token" to a system running in production, deployed by a pipeline, monitored honestly, and maintained by a team that can review each other's work and recover from its own mistakes. That's five tracks, thirteen chapters so far, built in dependency order on purpose — each one assumes the last. One track remains: chapter 15 wraps everything above in the security layer it needs before any of it touches a real customer.
The thread specific to this chapter: almost every default in the container and Kubernetes ecosystem assumes a stateless service that starts in seconds and costs little to run idle. None of that is true for a model server, and the sections above were built around exactly where that assumption breaks — startup probes, GPU scheduling, autoscaling metrics, weight versioning, and cost control that watches utilisation rather than the invoice.
What remains, honestly listed on the course index: JavaScript's own event loop and language fundamentals, MongoDB and DynamoDB at the operational depth chapter 12 deliberately stopped short of, and full system-design interview treatment. None of that is a missing foundation — it's genuine depth on top of a course that already gets someone from first principles to a production system, deployed safely, by a team that knows how to work together, and, after chapter 15, secured against the attack surface an LLM opens up.