Chapter 10 · Production serving
Building Agents with Google ADK
Chapter 8 argued that the agent loop is trivial and the harness is everything. This chapter builds that harness in one specific framework — grounded loops that can prove a claim, pipelines that compose four control-flow shapes, agents that choose their own next step, and a batch job that runs a multi-agent flow per row without one bad row taking down the run.
[!] What this chapter assumes, and what it adds
Section 8.25 already surveyed ADK: the agent types, the operational surface, A2A and agent cards, and an honest verdict on when adopting it is worth the coupling. This chapter does not repeat that survey. It assumes you have read it and have decided to build something.
The difference is depth and direction. 8.25 answers should I use this? This chapter answers how do I build the thing, and which of my instincts about it are wrong? Several are: an escalation that looks correct is silently discarded, a sub-agent is stickier than it appears, and a parallel branch cannot see what its sibling just produced. Each of those is demonstrated by running it.
[+] Everything here was executed against ADK 2.8.0
Agent frameworks move fast and their documentation lags, so no API in this chapter is quoted from memory. Every code block was run against a real google-adk 2.8.0 install, every claim about behaviour comes from a measurement, and where the measurement contradicted the intuitive answer, the measurement won and the surprise is called out. The numbers in the figures are the numbers those runs produced.
10.1 What ADK actually gives you
Strip away the branding and ADK offers one idea: control flow between agents should be structure in your code, not an instruction you hope a model follows.
It is worth being concrete about what that buys, because “orchestration framework” is vague enough to sound like it might be doing nothing. Consider a reviewer agent that should try at most three times to fix a draft. Written as a prompt, “attempt this at most three times” is a suggestion; a confused model that has lost track of its own history will happily attempt it nine times, and the failure mode is a bill rather than an error. Written as LoopAgent(max_iterations=3), four attempts is not a thing that can happen. The constraint holds precisely when the model is behaving worst, which is the only time constraints matter.
That is the same argument as 8.17's permission boundaries and 8.21's approval gates, applied to control flow instead of authority. It is also the honest limit of what a framework can do for you: ADK can guarantee how many times something runs and who is allowed to run it. It cannot guarantee that what runs is any good. The parts of this chapter that deal with quality — the validator, the entailment check, the ledger — are code you write, and the framework's role is only to give them a well-defined place to sit.
[def] The five things you will actually use
- LlmAgent
- A model, an instruction, a toolset. Everything else composes these.
- Session state
- A dictionary shared across the run. It is how agents pass work to each other, and the reason most pipelines need no glue code at all.
- Workflow agents
- SequentialAgent, ParallelAgent and LoopAgent — control flow as composition. See 10.16 for their deprecation status, which is genuinely awkward.
- Callbacks
- Hooks before and after an agent or a tool runs. This is where validation, redaction and budget enforcement live.
- Tools
- Plain Python functions, other agents wrapped as tools, or whole toolsets such as MCP servers and skills.
10.2 LlmAgent, and what state is for
The smallest useful agent is three fields and one that is easy to overlook.
from google.adk.agents import LlmAgent
# An LlmAgent is a model, an instruction and a toolset. Nothing more.
# `output_key` is the important part: whatever this agent returns is
# written to session state under that name, where later agents read it.
drafter = LlmAgent(
name="drafter", # must be unique among siblings
model="gemini-2.0-flash",
description="Writes a first draft of a company overview.",
instruction="Draft a two-sentence overview of the company.",
output_key="draft", # -> state["draft"]
)
output_key is the field that turns a collection of agents into a pipeline. Whatever the agent produces is written into the session's state dictionary under that name, and any later agent can read it by interpolating {draft} into its own instruction. There is no message passing to write and no return values to thread through — state is the bus.
[!] Two fields that look decorative and are not
- name must be unique among siblings, because it is the identifier used for transfers, for state namespacing, and in every trace you will ever read while debugging. Names like agent1 make a production trace unreadable.
- description is not documentation. When one agent chooses among several others, the description is what the model reads to decide. An agent with a great instruction and a vague description will be correct whenever it is chosen, and it will rarely be chosen.
The distinction between instruction and description catches people repeatedly, so it is worth stating as a rule: the instruction is read by this agent's model and says how to do the work; the description is read by other agents' models and says when to pick this one. They have different audiences and should not be copies of each other.
10.3 Testing without an API key
Before building anything larger, it is worth solving the problem that otherwise makes agent code untestable: every run costs money, takes seconds, and returns something slightly different. A test suite built on real model calls is slow, flaky, expensive, and cannot reproduce the specific failure you are trying to fix.
ADK only ever calls one method on a model, so satisfying that one method gives you a complete stand-in.
from typing import AsyncGenerator
from google.adk.models import BaseLlm, LlmResponse
from google.genai import types
class FakeLlm(BaseLlm):
"""A model that returns scripted text. No network, no API key.
Subclassing BaseLlm is the supported extension point: ADK only ever
calls generate_content_async, so anything satisfying this signature
is indistinguishable from Gemini to the rest of the framework.
"""
model: str = "fake"
replies: list[str] = []
async def generate_content_async(
self, llm_request, stream: bool = False
) -> AsyncGenerator[LlmResponse, None]:
text = self.replies.pop(0) if self.replies else "done"
yield LlmResponse(
content=types.Content(role="model", parts=[types.Part(text=text)])
)
[+] Why this is worth doing on day one
A scripted model lets you write the test you actually want: given a model that fabricates a revenue figure on its first attempt and corrects itself on the second, my pipeline must call the model exactly twice and emit the corrected version. That is a precise assertion about your harness, it runs in milliseconds, and it is impossible to write against a live model. Every behavioural claim in this chapter was verified this way — including the ones that turned out to be wrong.
10.4 Running an agent
A runner connects an agent tree to the services it needs — sessions, artifacts, memory. InMemoryRunner wires up in-memory versions of all of them, which is exactly right for tests and for development, and exactly wrong for production, where a restart would erase every conversation.
import asyncio
from google.adk.runners import InMemoryRunner
from google.genai import types
async def run_once(agent, prompt: str) -> dict:
"""Drive an agent to completion and return the final session state."""
runner = InMemoryRunner(agent=agent, app_name="demo")
# A session is one conversation. Its `state` dict is the channel every
# agent in the tree reads from and writes to.
session = await runner.session_service.create_session(
app_name="demo", user_id="u1"
)
# run_async yields every event as it happens: model output, tool calls,
# state updates. Draining it runs the agent to completion.
async for _ in runner.run_async(
user_id="u1",
session_id=session.id,
new_message=types.Content(role="user", parts=[types.Part(text=prompt)]),
):
pass
final = await runner.session_service.get_session(
app_name="demo", user_id="u1", session_id=session.id
)
return dict(final.state)
Two things in that snippet are worth dwelling on. First, run_async is an event stream rather than a function returning an answer: you receive model output, tool calls, and state changes as they happen, which is what makes progressive UI streaming (9.12) and live tracing possible. Draining it in a loop, as here, is simply the batch case.
Second, the final answer is read from session state, not from a return value. This feels indirect at first and pays off immediately: a pipeline of six agents needs no plumbing between them, and every intermediate result is still there afterwards for debugging, which is precisely what you want when the sixth agent produces something strange and you need to know which of the first five caused it.
10.5 Functions as tools
A tool in ADK is a plain Python function. There is no registration step and no schema to write, because the schema is derived from the signature and the docstring.
That convenience has a consequence worth internalising: the docstring is production code. It is not a note for the next developer, it is the text the model reads when deciding whether this function is the right one to call and what to pass it. Section 8.4 made this argument in general terms; ADK makes it literal, because there is no separate schema in which to say it better.
[!] Type hints are not optional here
An unannotated parameter gives the model no information about what to put in it, and the resulting argument will be a plausible-looking string a good deal of the time. A parameter typed order_id: str with a docstring line saying it is a 12-character identifier beginning with ORD- gets the right thing far more often. The general rule from 8.4 applies: make the wrong call hard to express, rather than detecting it afterwards.
10.6 Sub-agents versus AgentTool
This is the most consequential design decision in ADK, and the two options look almost identical when you write them. Both let one agent make use of another. What differs is who is in control afterwards — and the answer is stickier than most people expect.
from google.adk.agents import LlmAgent
from google.adk.tools import AgentTool
specialist = LlmAgent(
name="refunds",
model="gemini-2.0-flash",
description="Handles refund eligibility and processing.", # <- routing key
instruction="Decide refund eligibility from the order record.",
)
# OPTION A -- sub_agents: a TRANSFER, and it is sticky.
# Control moves to the specialist and stays there. The specialist answers
# the next user turn too. Right for "this conversation is now about
# refunds"; wrong if the parent still has work to do afterwards.
router = LlmAgent(
name="router",
model="gemini-2.0-flash",
instruction="Route the customer to the right specialist.",
sub_agents=[specialist],
)
# OPTION B -- AgentTool: a CALL, and it returns.
# The specialist runs, hands its answer back as a tool result, and the
# parent stays in control. Right when you need to combine several
# specialists, or do anything after the specialist finishes.
supervisor = LlmAgent(
name="supervisor",
model="gemini-2.0-flash",
instruction="Answer the customer, consulting specialists as needed.",
tools=[AgentTool(agent=specialist)],
)
Reading that code, the natural assumption is that the difference is mostly stylistic: one passes work down the tree, the other calls out sideways, and either way you get an answer back. That assumption is wrong, and it is worth seeing exactly how before it costs you an incident.
[def] Transfer is sticky
When a parent transfers to a sub-agent, control does not return at the end of the turn. The sub-agent becomes the active agent for the conversation and answers the next user message too, and the one after that, until something transfers control again.
Driven with a scripted model over two turns, the sub_agents version produced two replies both authored by specialist — the root never spoke again. The AgentTool version produced two replies both authored by root. Same specialist, same prompts, opposite ownership of the conversation.
This matters because the failure it produces is subtle rather than loud. A customer asks about a refund, the router transfers to the refunds specialist, the refund is handled correctly, and everyone is happy. Then the customer asks an unrelated question about delivery times — and the refunds specialist answers it, because it is still holding the conversation. It has no delivery tools and a refund-shaped instruction, so it produces something confident and unhelpful. Nothing errored, and no log line says anything went wrong.
[!] Transfers also cost you prompt caching
A detail ADK itself warns about at runtime, and one that does not appear in most tutorials: every transfer swaps the system instruction and the tool list, which changes the prefix of the request. Prompt caching keys on that prefix, so a transfer-heavy design re-sends the whole prompt uncached after each hop. On a chatty multi-agent system this is a real and recurring cost, not a rounding error. It is configurable — context_cache_config exists for exactly this — but the default is to pay it silently.
10.7 Choosing between them
The decision is not about hierarchy or code aesthetics. It is one question: after this other agent finishes, does the current agent still have work to do?
| Situation | Use | Why |
|---|---|---|
| The conversation genuinely changes subject and stays there | sub_agents | Stickiness is the desired behaviour, not a side effect |
| You need to combine results from two or more specialists | AgentTool | A transfer surrenders control, so there is nobody left to combine them |
| Anything must happen after the specialist finishes — formatting, logging, a follow-up question | AgentTool | Control returns, so “afterwards” exists |
| The specialist holds credentials the parent must never use | Either, but see 8.17 | The boundary is enforced by the toolset, not by the delegation style |
| You are unsure | AgentTool | Call-and-return is the less surprising default and is trivially reversible |
[retail] A support system that got this wrong
A triage agent routes to returns, delivery and billing as sub-agents. It works beautifully in testing, because every test is a single question. In production, the median conversation contains three questions on two topics, and the second question is answered by whichever specialist happened to handle the first. The fix is not more prompt engineering — it is AgentTool, so triage re-decides on every turn, which is what triage means.
10.8 Why prompts cannot enforce grounding
“Only use the provided sources and do not invent facts” is the most common instruction in production AI systems, and it is not an enforcement mechanism. It is a request, addressed to the component whose defining characteristic is that it sometimes produces confident text with no basis.
The gap between a request and an enforcement is easy to see once stated: if the model ignores the instruction, what in your system notices? For most deployments the honest answer is nothing, until a customer does. That is not a prompt-quality problem and no amount of rewording fixes it — the check has to exist outside the thing being checked.
[!] Provider-side search makes this structurally impossible
If you use a built-in grounded-search tool, the search happens inside the provider and your application receives generated prose that mentions sources. Your code never holds the source documents, so there is no text to compare a claim against. You cannot write the validator even if you want to. Owning retrieval — running the search yourself and keeping what came back — is what makes everything in this section possible, and it is a decision made long before the validator is written.
So the first component is not an agent at all. It is a record of what the run has actually seen.
from dataclasses import dataclass, field
@dataclass(frozen=True)
class Doc:
"""One retrieved document. `doc_id` is what a claim must cite."""
doc_id: str
url: str
text: str
@dataclass
class EvidenceLedger:
"""Every document the run has actually seen.
This is the piece that makes grounding enforceable rather than
aspirational. If the framework performs the search and hands the model
a summary, your code never holds the source text, so "do not invent
facts" can only ever be an instruction. Holding the documents yourself
turns it into an assertion you can run.
"""
docs: dict[str, Doc] = field(default_factory=dict)
def add(self, doc: Doc) -> None:
self.docs[doc.doc_id] = doc
def resolve(self, doc_id: str) -> Doc | None:
return self.docs.get(doc_id)
def corpus(self) -> str:
return "\n".join(d.text for d in self.docs.values())
The ledger is deliberately dull: a dictionary of documents and a way to resolve an identifier. Its value is entirely in existing at all, because it converts “the model should not invent facts” from a hope into a claim about two pieces of text that your code is holding.
10.9 A three-tier validator
The obvious way to check a draft is to ask a model whether it is supported by the evidence. That works, and if it is the only thing you do it is both slower and more expensive than necessary, because most fabrications can be caught with a regex.
Order the checks by cost. The cheap ones run on everything; the expensive one runs only on what survives.
import re
# A number in the draft that never appears in any retrieved document is the
# highest-signal fabrication there is: revenue figures, headcounts, dates.
# Checking it costs one regex and no tokens, so it runs on every claim.
_NUMERIC = re.compile(r"\d[\d,]*\.?\d*")
def normalise(token: str) -> str:
return token.replace(",", "").rstrip(".")
def validate_claim(claim: dict, ledger) -> list[str]:
"""Return a list of problems. Empty list means the claim is clean.
Tier 1 -- citation integrity: every cited doc_id must resolve. Catches
the model inventing a plausible-looking source.
Tier 2 -- literal grounding: every number in the text must appear in
at least one document the claim actually cites.
"""
problems: list[str] = []
cited = [ledger.resolve(i) for i in claim["doc_ids"]]
for doc_id, doc in zip(claim["doc_ids"], cited):
if doc is None:
problems.append(f"cites unknown document {doc_id}")
if not claim["doc_ids"]:
problems.append("claim has no citation")
supporting = " ".join(d.text for d in cited if d is not None)
for number in _NUMERIC.findall(claim["text"]):
if normalise(number) not in normalise(supporting):
problems.append(f"number {number} appears in no cited document")
return problems
[def] Why numbers are the highest-signal check
Fabricated prose is hard to detect mechanically, because paraphrase is legitimate: “the company is growing quickly” may be a fair reading of a document that never uses those words. Numbers are different. A revenue figure, a headcount, a date or a percentage either appears in the cited source or it does not, and paraphrase does not apply. Numbers are also the fabrications that do the most damage, because they are the parts a reader trusts most and are least likely to verify. One regex catches the highest-cost error class for no tokens at all.
Two details in that code are there because the naive version is wrong. Normalising away commas means 1,200 in the source matches 1200 in the draft — without it the validator produces constant false positives and the team learns to ignore it, which is worse than not having it. And the comparison is against the text of the cited documents only, not the whole corpus: a claim citing document B is not supported by a number that happens to appear in document D.
Then the expensive tier, for the failure the cheap ones cannot see.
async def entails(claim_text: str, evidence: str, judge) -> bool:
"""Tier 3: does the evidence actually support the prose?
Only reached for claims that already passed tiers 1 and 2, because it
is the only tier that costs a model call. A claim can cite a real
document and quote its numbers correctly and still misrepresent it --
"revenue grew" when the document says it fell -- and that is precisely
what this catches and the cheap tiers cannot.
"""
verdict = await judge(
"Answer only SUPPORTED or UNSUPPORTED.\n"
f"Evidence:\n{evidence}\n\nClaim: {claim_text}"
)
return verdict.strip().upper().startswith("SUPPORTED")
async def validate_all(claims: list[dict], ledger, judge) -> dict:
"""Run the cheap tiers on everything, the expensive tier on survivors."""
report: dict[str, list[str]] = {}
for claim in claims:
problems = validate_claim(claim, ledger)
if not problems:
evidence = " ".join(
ledger.resolve(i).text for i in claim["doc_ids"]
)
if not await entails(claim["text"], evidence, judge):
problems = ["evidence does not support this claim"]
if problems:
report[claim["text"]] = problems
return report
[+] What each tier catches that the others cannot
- Tier 1, citation integrity. The model invented a source, or cited nothing at all. Free.
- Tier 2, literal grounding. The source is real but the number is not. Free.
- Tier 3, entailment. Every citation resolves and every number is real, and the sentence still misrepresents the source — “revenue grew to $1.2 billion” drawn from a document reporting that revenue fell to $1.2 billion. Both cheap tiers pass this happily. Only a reader — human or model — catches it.
That last example is not hypothetical; it is the test case in the executed version of this code. The cheap tiers return no problems for it, and the judge rejects it. The same test also asserts the ordering works in the other direction: a claim with a fabricated number never reaches the judge at all, so the expensive tier is not spent on something already known to be broken.
10.10 The escalation that gets discarded
Here is the bug that motivates writing this section at all. It is invisible in review, it does not error, the output it produces is correct, and it multiplies your model spend by the loop bound.
A validator lives naturally in an after_agent_callback: the drafter runs, the callback inspects what it produced, and if everything checks out it asks the loop to stop. Asking the loop to stop means setting escalate. Both versions below do that, and only one works.
from google.adk.agents.callback_context import CallbackContext
def validator_BROKEN(callback_context: CallbackContext):
"""Looks correct. Does not stop the loop."""
callback_context.actions.escalate = True
return None
def validator_WORKING(callback_context: CallbackContext):
"""Identical intent, but the escalation actually survives.
ADK only turns the callback's actions into a real event if the callback
returns content or leaves a state delta. With neither, the escalate flag
is constructed and then dropped on the floor, and the loop runs to
max_iterations at full token cost -- while still producing correct
output, which is what makes it so easy to miss.
"""
callback_context.actions.escalate = True
callback_context.state["validation_passed"] = True # <- the load-bearing line
return None
[!] Measured, not theorised
With max_iterations=5 and a model whose very first draft is clean: setting escalate alone ran the drafter 5 times. Adding a single state write stopped it after 1. Identical intent, identical output, 5.0× the model calls.
The cause is in ADK's own handling of the callback. It only constructs an event carrying the callback's actions if the callback returned content or left a state delta. With neither, the escalate flag is set on an object that is then discarded, so the loop never learns anything happened. The reason this survives review is that the output is still correct — the extra iterations regenerate an already-good draft, the final answer looks right, and the only symptom is the invoice.
[def] A regression test must fail when you break it
This bug is a good argument for mutation testing, and it earned that reputation honestly: an earlier version of this material described the rule as “the callback must return content,” which is only half true. That error was caught by deliberately reintroducing the bug and discovering the test still passed — it was accidentally protected by an unrelated state write elsewhere. If you write a test for this, break the code on purpose and confirm the test goes red. A test that passes either way is not a test.
There is also a way to avoid the trap entirely. ADK ships the escape hatch as a tool.
from google.adk.tools import exit_loop
# ADK ships the escape hatch as a tool. Given to an LlmAgent inside a
# LoopAgent, the model calls it when its own instruction says the work is
# done, and the loop stops. Because a tool call is already a real event,
# the escalation is never discarded -- the failure mode above cannot occur.
reviewer = LlmAgent(
name="reviewer",
model="gemini-2.0-flash",
instruction=(
"Review the draft in {draft}. If every claim is supported, call "
"exit_loop. Otherwise reply with the specific problems to fix."
),
tools=[exit_loop],
)
Because a tool call is already a real event, an escalation raised from tool context cannot be discarded — the failure mode above simply does not apply. It also moves the stop decision into the model's reasoning, where it is visible in traces as an explicit action rather than inferred from a callback's side effects. Prefer this when the stopping criterion is a judgement call. Keep the callback approach when the criterion is deterministic, as it is for the validator, since you do not want to ask a model whether the regex matched.
10.11 The repair loop
With a validator and a working escalation, the loop writes itself — but one detail separates a repair loop from an expensive retry.
from google.adk.agents import LlmAgent, LoopAgent
from google.adk.agents.callback_context import CallbackContext
def make_grounding_gate(ledger):
"""after_agent_callback that stops the loop only when the draft is clean.
Note both halves of the escape: `escalate` states the intent, and the
state write is what makes ADK materialise an event carrying it. Setting
only the flag leaves the loop running to max_iterations.
"""
def gate(callback_context: CallbackContext):
draft = callback_context.state.get("draft", "")
claims = parse_claims(draft)
problems = {
c["text"]: p
for c in claims
if (p := validate_claim(c, ledger))
}
if not problems:
callback_context.actions.escalate = True
callback_context.state["grounded"] = True # makes escalate stick
return None
# Feed the specific failures back so the next pass is a repair
# rather than a reroll. The drafter reads {problems} in its prompt.
callback_context.state["problems"] = problems
return None
return gate
def build_grounded_loop(model, ledger, max_iterations: int = 3) -> LoopAgent:
drafter = LlmAgent(
name="drafter",
model=model,
instruction=(
"Write the overview using only the supplied evidence. Cite a "
"doc_id for every claim. Problems to fix: {problems?}"
),
output_key="draft",
after_agent_callback=make_grounding_gate(ledger),
)
# max_iterations is a structural bound, not a request. The loop cannot
# exceed it however confused the model becomes.
return LoopAgent(name="grounded", sub_agents=[drafter],
max_iterations=max_iterations)
[+] Feed back the specific failure, not the fact of failure
Writing the problems into state and interpolating them into the drafter's instruction is what makes the second attempt a repair. Simply re-running the drafter is a reroll: the same prompt, the same evidence, and a fresh sample from the same distribution that just produced a fabrication. Telling it exactly which number was unsupported changes the input, and therefore changes the odds. This is the same principle as 8.7's structured tool errors — an error message is only useful if it says what to do differently.
The executed version of this loop asserts three behaviours: a clean first draft stops after one pass; a draft with a fabricated number is caught and the corrected second draft ends the loop after exactly two; and a model that never gets it right terminates at max_iterations rather than running forever. That third case is the one worth testing deliberately, because it is the one that only shows up in production, on the input nobody thought about.
10.12 Stall detection and quarantine
A bounded loop cannot run forever, but it can still waste its entire budget usefully doing nothing. Two refinements are worth the small amount of code they cost.
Stall detection. If the set of problems after this pass is identical to the set before it, the model is not converging — it is producing the same failure with different wording. Three iterations of that is three times the cost of one, with the same outcome. Compare the problem set between passes and stop early when it stops shrinking; you have learned everything the loop is going to teach you.
Quarantine. When the loop exhausts its budget with problems outstanding, the question is what to publish. Publishing the draft with its unsupported claims intact is the worst option and, unfortunately, the default if nobody decides otherwise. The better move is to drop the specific claims that failed and emit the rest, marked as incomplete — the section that says “revenue: not found” is far more useful than the one that invents a number, and it is the behaviour 5.31's abstention discussion argues for.
[!] Make quarantine monotonic
Once a claim has failed validation, it should stay out. A subsequent pass that reproduces the same unsupported sentence must not be able to reintroduce it by happening to phrase it in a way the validator does not catch — otherwise the loop's last iteration can undo the work of all the earlier ones. Track rejected claims across iterations, not within one.
10.13 Sequential, parallel, loop
Three control-flow shapes, each corresponding to a property of the work rather than a preference about code. Choosing between them is a question about your task, not about the framework.
| Shape | Use when | The property being exploited |
|---|---|---|
| SequentialAgent | Each stage needs the previous stage's output | Dependency. There is no choice, and no latency to win back. |
| ParallelAgent | Sub-tasks are genuinely independent | Independence. Latency becomes the slowest leg rather than the sum. |
| LoopAgent | Quality is reached by iteration, not in one pass | Convergence — with a structural cap, since convergence is not guaranteed. |
The measurement is worth stating plainly, because “parallel” in an async framework is not always as parallel as advertised. Three legs of 0.4s each, run through a ParallelAgent, completed in 0.41s against 1.2s if run one after another — a 3.0× speedup, which is the whole of the theoretical maximum. The branches genuinely overlap.
[!] A measurement I got wrong first time
The first version of that experiment reported 2.02s for two 0.5s legs and I briefly concluded ADK was running them serially. It was not — the harness was measuring setup it should not have been. Re-running it with per-call timestamps showed both branches starting at the same instant and ending at the same instant. The lesson is the one from chapter 19: a single wall-clock number is a weak measurement, and the fix is to instrument the thing you actually care about rather than time the whole program.
10.14 What parallel branches can see
Parallel sub-agents run in isolated branches, and the consequence is the thing people get wrong: a branch cannot read what a sibling produced during the same parallel block. If one leg writes findings_people and another leg's instruction interpolates {findings_people}, that placeholder resolves to nothing.
[def] Isolated during, merged after
Every branch's writes do land in session state, and after the parallel block completes they are all visible to whatever runs next. So the pattern that works is fan-out then join: parallel legs gather independently, and a following sequential stage reads all of their outputs. The pattern that silently produces empty strings is one leg trying to build on another leg's work. If leg B needs leg A's output, they are not independent, and they should not be in the same parallel block.
This is worth checking against your intuition, because the failure is quiet. There is no error and no warning; the interpolation resolves to an empty string, the model writes something plausible around the gap, and the result reads fine. It is the same class of problem as the discarded escalation — correct-looking output hiding a structural mistake.
10.15 A full composed pipeline
Now the scenario that combines everything: a research report that plans its own work, gathers evidence on several fronts at once, drafts and re-drafts under validation, and assembles a final document. Four agents, three control-flow shapes, one validator.
from google.adk.agents import LlmAgent, LoopAgent, ParallelAgent, SequentialAgent
def build_pipeline(model, ledger) -> SequentialAgent:
"""Plan -> (research x N in parallel) -> draft/critique loop -> assemble.
Each shape is chosen for a property, not for variety:
Sequential -- later stages need earlier stages' output.
Parallel -- the research legs are independent, so latency is one
leg rather than the sum. Branches are isolated: a
sibling's output_key is NOT visible mid-run.
Loop -- quality is a fixed point, not a single pass, and the
iteration cap is structural.
"""
planner = LlmAgent(
name="planner", model=model, output_key="plan",
instruction="Break the request into research questions.",
)
legs = [
LlmAgent(name=f"research_{topic}", model=model,
output_key=f"findings_{topic}",
instruction=f"Research {topic} for: {{plan}}")
for topic in ("financials", "people", "risks")
]
research = ParallelAgent(name="research", sub_agents=legs)
drafter = LlmAgent(
name="drafter", model=model, output_key="draft",
instruction=(
"Draft from {findings_financials}, {findings_people}, "
"{findings_risks}. Fix any problems in {problems?}."
),
after_agent_callback=make_grounding_gate(ledger),
)
refine = LoopAgent(name="refine", sub_agents=[drafter], max_iterations=3)
assembler = LlmAgent(
name="assembler", model=model, output_key="report",
instruction="Format {draft} as the final report.",
)
return SequentialAgent(
name="pipeline", sub_agents=[planner, research, refine, assembler]
)
[+] Read the structure, not the prompts
The valuable property of this code is that you can determine what will happen without reading a single instruction string. Planning happens once. Research happens three ways at once. Drafting happens between one and three times. Assembly happens last, and exactly once. None of that is inferable from a prompt-driven design, where the same behaviour would be a paragraph of English and a hope. This is what 8.25 meant by structural orchestration, and it is the strongest argument for a framework like this one.
Note also what the shapes cost. The parallel block turns three sequential model calls into one wall-clock leg, which is the difference between a report that takes ninety seconds and one that takes half a minute. The loop's bound means a pathological input costs at most three drafts instead of an unbounded number. Both are properties of the structure, and neither depends on the model cooperating.
[retail] Where the seams go in a real system
A supplier-risk report at a retailer looks exactly like this: plan the questions, then hit financial filings, news, and internal incident history simultaneously — three independent sources, no reason to serialise them — then draft under a validator that refuses any risk claim without a resolvable citation, because this document informs a purchasing decision and an invented risk is as damaging as a missed one. The assembler exists so the output format is decided in one place rather than being something each drafting pass argues with itself about.
10.16 The deprecation problem
An honest note, because this will confuse anyone reading the source. In ADK 2.8.0 all three workflow agents — SequentialAgent, ParallelAgent and LoopAgent — are marked deprecated in favour of a newer graph-based Workflow API. Importing them emits a DeprecationWarning.
[!] Deprecated, and still the only option
The deprecation notice carries its own caveat: “Workflow cannot yet be used as an LlmAgent sub-agent.” So if you need a loop or a parallel block nested inside an agent — which is what every example in this chapter does — the deprecated classes remain the only way to express it. Workflow is also not exported from google.adk.agents yet; it lives in a private module.
The practical response is not to avoid the deprecated classes, which would mean not building the thing. It is to confine them. Keep every construction of a workflow agent inside one small module that exposes your own pipeline-building functions, as build_pipeline does above. Then the migration, when Workflow is ready, is one file rather than a search across the codebase — and your business logic, which does not care how the loop is implemented, never mentions the deprecated names at all.
This is a specific instance of a general habit worth having with fast-moving frameworks: the parts of a young library most likely to be rewritten are its orchestration primitives, and the parts least likely are plain functions and data. Keep your value in the latter. It is also the same advice 8.25 gives about ADK as a whole — use the operational surface, but keep your tool implementations and business rules in ordinary Python that would survive changing frameworks entirely.
10.17 Is “skills” a Claude word?
It is a reasonable suspicion. Agent skills arrived with a great deal of Anthropic branding, and vendor-specific vocabulary dressed up as an industry standard is common enough that scepticism is the right default. In this case the scepticism is misplaced, and the way to settle it is to look in the package rather than at the marketing.
Google ADK 2.8.0 ships a google.adk.skills module. It is not a compatibility shim or a thin adapter — it contains the data model (Skill, Frontmatter, Resources, Script), a registry, loaders for both local directories and cloud storage, and a SkillToolset that exposes skills to a model. Loading a SKILL.md file written to the Claude conventions into an ADK agent works without modification.
[+] The decisive evidence is in the source
ADK's own docstring for the allowed_tools field points the reader at agentskills.io/specification — a vendor-neutral specification, not Anthropic's documentation. A framework does not cite an external standard for a format it considers proprietary to a competitor. Skills are a cross-vendor convention that Anthropic originated and published; ADK implements it.
So the accurate framing is the same one 8.9 used for MCP and 8.25 used for A2A. All three are protocols that began at one vendor and became shared, and their value comes precisely from being implemented in more than one place. Skills are the third member of that family: MCP standardises how an agent reaches a tool, A2A how it reaches another agent, and skills how it acquires instructions.
10.18 Writing and loading a skill
A skill is a folder, not a Python object. That is the entire idea, and it is what makes skills portable between frameworks in a way that a class hierarchy never could be.
skills/
refund-policy/
SKILL.md # required: YAML frontmatter + markdown instructions
references/ # optional: deeper documentation, loaded on demand
assets/ # optional: templates and other resources
scripts/ # optional: executables the agent may run
The frontmatter carries the metadata; the body carries the instructions.
---
name: refund-policy
description: Apply the refund policy. Use when a customer asks for a
refund, a return, or an exchange.
license: Apache-2.0
metadata:
adk_inject_state: true
---
# Refund policy
The customer's loyalty tier is {tier?}.
1. Within 30 days of delivery, with proof of purchase, issue a full refund.
2. Between 30 and 90 days, issue store credit.
3. After 90 days, no refund. Offer a repair booking instead.
Loading it is two lines.
from google.adk.agents import LlmAgent
from google.adk.skills import load_skills_from_dir
from google.adk.tools.skill_toolset import SkillToolset
# A skill is a folder on disk, not a Python object:
# skills/refund-policy/SKILL.md <- YAML frontmatter + instructions
# skills/refund-policy/references/ <- optional deeper docs
# skills/refund-policy/scripts/ <- optional executables
#
# Only each skill's name and description are put in the prompt up front.
# The body is loaded on demand, when the model decides the skill is
# relevant -- which is the whole point: you can ship fifty skills without
# paying fifty skills' worth of context on every turn.
skills = load_skills_from_dir(SKILLS_DIR)
agent = LlmAgent(
name="support",
model="gemini-2.0-flash",
instruction="Help the customer. Use a skill when one applies.",
tools=[SkillToolset(skills=skills)],
)
[def] Progressive disclosure is the whole point
Only each skill's name and description are placed in the prompt up front. The body is fetched only when the model decides the skill is relevant and calls load_skill. That is what makes fifty skills affordable: you pay fifty short descriptions per turn instead of fifty full documents. The executed version of the snippet above asserts this directly — it confirms the phrase “store credit” is present in the skill body and absent from the block sent up front.
Two ADK-specific extensions live in metadata, and both are worth knowing. adk_inject_state: true enables {variable} interpolation in the skill body from session state — so the refund policy above can address the customer's actual loyalty tier, with {tier?} resolving to an empty string rather than raising when the key is missing. adk_additional_tools lets a skill declare tools that become available when it is loaded.
[!] The description is a routing decision, again
As with an agent's description in 10.2, this is the field that determines whether the skill is ever used. “Refund policy” is a label; “Apply the refund policy. Use when a customer asks for a refund, a return, or an exchange” is a routing instruction that names the trigger conditions. ADK enforces a 1024-character limit here, which is generous — the constraint is not length, it is remembering that this text has a job to do.
10.19 Skills versus tools versus agents
Three mechanisms for extending what an agent can do, and a reliable way to choose: what are you actually adding?
| You are adding | Use | Because |
|---|---|---|
| A capability — something the model cannot do itself, like querying a database or sending an email | A tool | Code executes; the model only decides when |
| Knowledge or procedure — a policy, a house style, a multi-step checklist | A skill | It is instructions, and it is too long to keep in the prompt permanently |
| A separate reasoning context — different model, different tools, different authority | An agent | You need isolation, not just extra text or one function |
[+] A useful test when two of them seem to fit
Ask what would happen if you pasted the content into the system prompt permanently. If the answer is “that works, it is just wasteful” — it is a skill, and the win is context economy. If the answer is “that is impossible, it needs to run code” — it is a tool. If the answer is “that would conflict with the instructions already there” — it is an agent, because what you actually need is a separate context, and this is 8.17's argument restated.
It is worth noting that skills partly close the gap that made 8.14's context engineering so difficult. The old dilemma was between a bloated system prompt that covered every eventuality and a lean one that failed on unusual cases. Skills reframe it: keep the system prompt lean, keep the long-tail procedures on disk, and let the model fetch what the current situation calls for. That is retrieval — applied to instructions rather than documents, which is a genuinely good idea and slightly overdue.
10.20 Fixed pipelines versus autonomy
Everything so far has had its control flow decided in advance. 10.15's pipeline always plans, then researches, then drafts, then assembles. The model chooses the words; the code chooses the sequence. A fully autonomous agent inverts that: the model chooses the sequence too.
The temptation is to treat autonomy as the more advanced option and fixed pipelines as the beginner's version. That gets the trade backwards. Autonomy is what you resort to when you cannot specify the sequence, and it costs you every guarantee that came from specifying it.
| Property | Fixed pipeline | Autonomous agent |
|---|---|---|
| Cost per run | Predictable within a narrow range | Varies by an order of magnitude between inputs |
| Latency | Bounded by the structure | Bounded only by the step cap |
| Debugging a bad run | Find the stage that failed | Reconstruct a sequence of decisions that will not recur |
| Handles tasks whose shape is unknown in advance | No — it does what it was built to do | Yes, and this is the entire reason to accept the above |
| Safe with side-effecting tools | Auditable up front | Requires every control in 8.20–8.22, without exception |
[!] The honest default
Most tasks presented as needing an autonomous agent are a fixed pipeline that nobody has sat down and specified. “Handle customer emails” sounds open-ended and is usually classify, retrieve, draft, check — four stages, known in advance, cheaper and more debuggable as a SequentialAgent. Reach for autonomy when the task genuinely branches on what earlier steps discover, and 17.22's warning about complexity you cannot measure applies directly.
10.21 Building an autonomous agent
The construction is simpler than the fixed pipeline, which is itself a useful signal about where the difficulty has moved.
from google.adk.agents import LlmAgent, LoopAgent
from google.adk.tools import AgentTool, exit_loop
def build_autonomous(model, tools, specialists, max_steps: int = 12) -> LoopAgent:
"""A fully dynamic agent: it decides its own next step each iteration.
The difference from the composed pipeline is where control flow lives.
There, the sequence was fixed in code and the model filled in content.
Here the model chooses the sequence, and the *bounds* are what live in
code. That trade buys flexibility on tasks whose shape is not known in
advance, and it costs you the ability to predict what will run.
Note what remains structural even at full autonomy:
- max_steps caps the loop no matter what the model believes;
- specialists are AgentTools, so control always returns here;
- exit_loop is the only way to finish early, and it is explicit.
"""
worker = LlmAgent(
name="worker",
model=model,
instruction=(
"Goal: {goal}\n"
"Progress so far: {scratchpad?}\n"
"Take ONE step towards the goal. Append what you learned to the "
"scratchpad. When the goal is fully met, call exit_loop."
),
tools=[*tools, exit_loop, *(AgentTool(agent=s) for s in specialists)],
output_key="scratchpad",
)
return LoopAgent(name="autonomous", sub_agents=[worker],
max_iterations=max_steps)
One agent, in a loop, with a scratchpad. Each iteration it reads the goal and what it has learned, takes one step, and appends the result. It stops when it decides the goal is met and calls exit_loop, or when the step cap is reached. This is 8.3's agent loop and 8.11's planning discussion, expressed in ADK's primitives.
[+] The specialists are tools, deliberately
Note that specialists are wrapped as AgentTool rather than attached as sub-agents. In an autonomous loop this is not a stylistic choice. A transfer would hand control away permanently, and the loop — along with its step cap and its scratchpad — would simply stop being in charge. Wrapping them as tools means the loop always regains control, so its bounds continue to apply. Autonomy is exactly the setting where the sticky-transfer behaviour from 10.6 does the most damage.
10.22 Bounding what you cannot predict
Since you can no longer say what an autonomous agent will do, everything rests on what it cannot do. Four bounds, in decreasing order of how much they save you.
A step cap. max_iterations is the difference between a bad run costing a few dollars and a bad run costing until someone notices. Set it from a measurement — the 95th percentile of successful runs on your evaluation set, plus headroom — rather than from a round number that feels generous.
A spend cap. Steps are a poor proxy for cost, because one step that reads a 200-page document costs more than twenty that do not. Track tokens in a callback and stop on the budget, as 8.22 argues. An agent that has spent its allowance should stop even if it has steps remaining.
Read-only by default. The single highest-leverage control. An autonomous agent with only read tools has a worst case of wasting money and returning something wrong. Add one side-effecting tool and the worst case becomes unbounded external damage. Every write should be a separate, approved, idempotent tool — 8.5 and 8.21 in full, applied without exception.
Stall detection. Same idea as 10.12, and more important here. An autonomous agent that has stopped making progress will keep looking busy: it will re-read the same page, rephrase the same query, and consume its entire budget confidently. Compare the scratchpad between iterations and stop when it stops growing in substance.
from google.adk.agents import LlmAgent
from google.adk.workflow._retry_config import RetryConfig
# Per-agent retry and timeout are built in -- there is no need to wrap
# agents in your own retry decorator, and doing so tends to multiply with
# this one rather than replace it.
researcher = LlmAgent(
name="researcher",
model="gemini-2.0-flash",
instruction="Research the question.",
timeout=60.0, # seconds, per agent run
retry_config=RetryConfig(
max_attempts=3,
initial_delay=1.0,
backoff_factor=2.0, # 1s, 2s, 4s
jitter=0.3, # avoids a synchronised retry storm
),
)
[def] Use the built-in retry, not your own
timeout and retry_config are fields on every agent, so transient failures do not need a decorator you wrote. This matters more than it sounds: a hand-rolled retry wrapped around an agent that is already retrying multiplies rather than replaces, and three attempts times three attempts is nine model calls for one logical step. Jitter is worth setting for the reason 16.14 gives — without it, a batch of agents that fail together will retry together and reproduce the overload that caused the failure.
10.23 One flow per row
A different shape of problem: not one conversation with a user, but a hundred thousand rows in a table, each of which must trigger its own multi-agent flow. Enrich every supplier record. Classify every support ticket from last quarter. Draft a summary for every contract. The agent design is the same; everything around it changes.
The first decision is the one that determines whether the rest works, and it is easy to get wrong in a way that only shows up at scale: each row gets its own session.
import asyncio
from google.adk.runners import InMemoryRunner
from google.genai import types
async def process_row(row: dict, build_agent, limit: asyncio.Semaphore) -> dict:
"""Run the full multi-agent flow for exactly one row.
Three properties matter here, and each is one line:
- a session PER ROW, so rows cannot see each other's state;
- a semaphore, so 100k rows do not open 100k concurrent model calls;
- a try/except returning a value, so one bad row is a recorded
failure rather than a dead batch.
"""
async with limit:
runner = InMemoryRunner(agent=build_agent(), app_name="batch")
session = await runner.session_service.create_session(
app_name="batch",
user_id=row["id"], # isolation key
state={"row": row}, # seed input, no prompt stuffing
)
try:
async for _ in runner.run_async(
user_id=row["id"],
session_id=session.id,
new_message=types.Content(
role="user", parts=[types.Part(text=row["text"])]
),
):
pass
except Exception as exc: # noqa: BLE001 -- deliberate
return {"id": row["id"], "status": "failed", "error": str(exc)}
final = await runner.session_service.get_session(
app_name="batch", user_id=row["id"], session_id=session.id
)
return {"id": row["id"], "status": "ok", "state": dict(final.state)}
async def run_batch(rows, build_agent, concurrency: int = 8) -> list[dict]:
limit = asyncio.Semaphore(concurrency)
# return_exceptions=True is the difference between "one row raised" and
# "the batch died on row 40,000 after four hours".
return await asyncio.gather(
*(process_row(r, build_agent, limit) for r in rows),
return_exceptions=True,
)
[!] Reusing one session across rows is the classic bug
It is tempting, since creating a session per row feels wasteful. What actually happens is that row 2 inherits row 1's conversation history and state. The model, quite reasonably, treats it as context — and starts producing summaries of supplier B that mention supplier A. Because it is a correctness failure rather than a crash, it is discovered by whoever reads the output, which at batch scale is often nobody. A session is cheap; contamination is not.
Note also how the row's data enters: as session state, not by formatting it into the prompt. This keeps structured input structured, avoids an entire class of injection problems from row content that happens to contain instruction-like text (15.7), and means any agent in the tree can read the original row rather than the model's paraphrase of it.
10.24 Concurrency, failure, and cost
Three properties, each worth one line of code, and each catastrophic to omit.
Bound the concurrency. asyncio.gather over 100,000 rows creates 100,000 coroutines that all try to call the model at once. You will hit rate limits within seconds, and the retry storm that follows will make it worse. A semaphore bounds in-flight work regardless of batch size. Verified: with a limit of 2 and six rows, peak concurrency was exactly 2.
Isolate the failures. One row that raises must not end the run. return_exceptions=True plus a per-row try turns a fatal error into a recorded one. Verified: with six rows and one deliberately raising, five completed and the failure was captured with its row identifier intact.
Know the cost before you start. A multi-agent flow with a plan stage, three parallel research legs and a loop that may run three times is not one model call per row. Count the calls in the worst case, multiply by the row count, and price it before running the job rather than after.
[def] The arithmetic that ends most batch plans
10.15's pipeline is 1 planner + 3 research + up to 3 drafts + 1 assembler = up to 8 model calls per row. Across 100,000 rows that is 800,000 calls. At two seconds each with a concurrency of 8, that is roughly 55 hours of wall-clock time. This calculation takes a minute and routinely changes the design — usually by finding rows that do not need the full pipeline, which is the next point.
[+] Filter before the agent, not inside it
The cheapest model call is the one that does not happen. If 60% of rows can be resolved by a rule, a lookup, or a classifier costing a thousandth as much, run that first and send only the remainder to the agent. Using a multi-agent pipeline to decide that a row is uninteresting is the most expensive possible way to reach that conclusion, and it is a common design because the agent is the new tool and everything looks like a nail.
10.25 Checkpointing and dead letters
A batch job long enough to be worth parallelising is long enough to be interrupted. The difference between a four-hour job that can resume and one that cannot is a few lines.
import asyncio
import json
async def run_batch_streaming(rows, build_agent, out_path, dead_path,
concurrency: int = 8) -> dict:
"""Write each result as it lands, so a crash at row 90,000 is not fatal.
Collecting into a list and writing at the end means an OOM or a killed
pod loses the whole run. Appending per row means a rerun skips what is
already done. For a long batch this is the difference between a retry
costing minutes and costing the entire job again.
"""
limit = asyncio.Semaphore(concurrency)
tally = {"ok": 0, "failed": 0}
lock = asyncio.Lock()
with open(out_path, "a") as out, open(dead_path, "a") as dead:
async def one(row):
result = await process_row(row, build_agent, limit)
async with lock: # one writer at a time
if result["status"] == "ok":
out.write(json.dumps(result) + "\n")
tally["ok"] += 1
else:
# The dead-letter file is the retry queue: it holds the
# row itself, not just the error, so a rerun is trivial.
dead.write(json.dumps({"row": row, **result}) + "\n")
tally["failed"] += 1
await asyncio.gather(*(one(r) for r in rows))
return tally
[+] Two files, two purposes
- Results, appended per row. Collecting into a list and writing at the end means a crash at row 90,000 loses everything. Appending as you go means a rerun skips what is already done. At batch scale this is the difference between a retry costing minutes and costing the whole job.
- A dead-letter file, holding the row itself. Not just the error — the input. That makes the retry trivial: filter the rows that failed, fix the cause, and resubmit exactly those. A dead-letter file containing only stack traces tells you what broke and leaves you to reconstruct what to re-run.
The single lock around the writes is doing real work, and it is the kind of detail that is invisible until it is not. Without it, concurrent coroutines interleave their writes and produce lines that are half one JSON object and half another — a file that looks fine until something tries to parse it, which is usually the following morning.
[retail] A supplier-enrichment job, end to end
80,000 suppliers, each needing the 10.15 pipeline. The shape that works: a cheap pre-filter drops the 40% with nothing to enrich; a semaphore of 8 keeps the model within its rate limit; each surviving row gets its own session seeded with the supplier record; results append to JSONL as they land; failures go to a dead-letter file with the row attached. When it dies at hour three because a credential expired, the rerun skips 30,000 completed rows and costs twenty minutes instead of three hours. Every one of those decisions is boring, and together they are the difference between a job that ships and a job that gets abandoned after the second failed overnight run.
[!] Sample before you commit
Run 100 rows and read the output yourself before launching 80,000. Not the pass rate — the actual text. Systematic problems that no validator catches, like every summary opening with the same clause or the model consistently misreading one column, are obvious in ten samples and invisible in an aggregate score. This is 8.26's evaluation argument at batch scale, and the sampling costs a rounding error of the full run.
10.26 Key takeaways
The framework gives you structure. Everything that determines whether the output can be trusted is code you write and place inside that structure.
| Decision | The rule | Why |
|---|---|---|
| Sub-agent or tool | Default to AgentTool | Transfer is sticky — the specialist keeps answering later turns |
| Stopping a loop from a callback | escalate and a state write, or use exit_loop | The flag alone is discarded; measured at 5.0× the model calls |
| Enforcing grounding | Own retrieval, keep a ledger, validate in tiers | A prompt cannot check itself; provider-side search makes checking impossible |
| Validator ordering | Citations, then numbers, then entailment | Two of the three are free; only spend tokens on what survives them |
| Parallel branches | Fan out, then join — never chain within a block | Siblings cannot see each other's output mid-run, and fail silently |
| Workflow agents | Use them, but confine them to one module | Deprecated, yet still the only way to nest under an LlmAgent |
| Skills, tools or agents | Instructions, capability, or isolation — pick by what you are adding | Each solves a different problem; the overlap is only apparent |
| Autonomy | Only when the sequence genuinely cannot be specified | You trade every cost, latency and debugging guarantee for flexibility |
| Batch | Session per row, bounded concurrency, dead letters | Contamination, rate limits and lost work are the three failure modes |
[+] The pattern behind the three worst bugs here
The discarded escalation, the sibling that reads an empty string, and the reused batch session share a signature: the output still looks correct. Nothing raises, nothing logs a warning, and the result passes a glance. That is what makes them expensive — they are found by an invoice, an auditor, or a customer rather than by a test. When working with a framework whose whole job is to hide control flow from you, the failures worth fearing are the quiet ones, and the only reliable defence is to assert on behaviour you have actually measured.
[def] What is genuinely portable from this chapter
Almost all of it. The evidence ledger, the tiered validator, the repair loop with specific feedback, stall detection, monotonic quarantine, session-per-row, bounded concurrency, dead-letter queues — none of these are ADK ideas. They are the answers to problems that any agent framework leaves for you, and they transfer intact to LangGraph, CrewAI, or two hundred lines of your own async code. What ADK contributes is a tidy place to put them. Keep them in plain functions, as 8.25 recommends, and the framework becomes a decision you can revisit rather than one you are married to.
10.27 Interview drills
Framework questions are a trap in both directions. Reciting an API is not evidence of anything, and dismissing frameworks entirely usually means never having operated one. What an interviewer is listening for is whether you know which guarantees are structural and which are wishful.
1. What is the difference between adding an agent as a sub-agent and wrapping it as a tool?
A sub-agent is a transfer and a tool is a call. That sounds like a distinction about code structure, but the consequence is about conversation ownership: after a transfer, the sub-agent stays active and answers the next user turn too. With AgentTool, the specialist returns a result and the parent remains in control.
I check this by running two turns rather than one. A single-turn test cannot tell the two apart, which is exactly why the bug reaches production — every test asks one question, and real conversations do not.
My default is AgentTool, because it composes: I can consult two specialists and combine them, or do anything at all afterwards. I use sub-agents when stickiness is genuinely what I want — the conversation has changed subject and should stay there. There is also a cost angle: each transfer swaps the system prompt and toolset, which breaks prompt caching and re-sends the whole prefix uncached.
2. Your validator sets escalate on the callback context, and the loop still runs to max_iterations. Why?
Because the flag only reaches the loop if the callback also returns content or writes to state. ADK constructs the event carrying those actions in one of those two cases; with neither, the escalation is set on an object that is then thrown away. The loop never hears about it.
What makes this nasty is that nothing looks broken. The extra iterations regenerate an already-acceptable draft, so the output is fine and the only symptom is spend. I measured it at five iterations instead of one on a five-bound loop — five times the model calls for an identical result.
Two fixes. Either write something to state alongside the flag, which is what I do when the stopping condition is deterministic; or use the exit_loop tool, which sets the same flag from tool context where it cannot be discarded. And I would test it by reintroducing the bug deliberately to confirm the test actually fails — I have been caught by a test that passed either way because an unrelated state write was protecting it.
3. How would you stop an agent from fabricating figures in a generated report?
Not with a prompt. “Only use the provided sources” is a request to the component that fabricates, and nothing in the system notices when it is ignored.
The precondition is owning retrieval. If I use a provider's built-in grounded search, my application never holds the source documents, so I have nothing to compare a claim against and the check is impossible to write. So: run the search myself, keep every document in a ledger keyed by an identifier, and require each claim to cite identifiers.
Then validate in three tiers, ordered by cost. Citation integrity — does every cited id resolve? Literal grounding — does every number in the claim appear in the documents that claim actually cites? Both are regex-cheap and run on everything. Only what survives goes to an entailment check with a model.
Numbers get their own tier because they are the highest-signal and highest-damage case. Paraphrase makes prose hard to check mechanically, but a revenue figure either appears in the source or it does not — and it is the part a reader trusts most and verifies least.
4. Why bother with an entailment check if the cheap checks already passed?
Because a claim can be perfectly cited, contain only real numbers, and still say the opposite of its source. “Revenue grew to $1.2 billion” drawn from a document reporting that revenue fell to $1.2 billion passes both cheap tiers — the citation resolves and the number is genuine. Only reading it catches the reversal.
That is also the argument for ordering rather than choosing. Entailment is the only tier that costs a model call, so it runs last and only on survivors. A claim with a fabricated number is already known to be broken; sending it to a judge is spending money to re-learn something a regex established for free.
5. Your loop hits its iteration cap with problems outstanding. What do you publish?
Not the draft with its unsupported claims intact, which is unfortunately the default if nobody decides otherwise. I drop the specific claims that failed and publish the rest, marked incomplete. A section reading “revenue: not found” is more useful than one containing an invented number, because a reader can act on a gap and cannot act on a plausible falsehood.
I would also make the quarantine monotonic. Once a claim has failed, it stays out — otherwise a later pass can reintroduce it by rephrasing it in a way the validator happens to miss, and the final iteration undoes the work of all the earlier ones.
Before that, I would add stall detection. If the problem set is identical between two passes, the model is not converging and the remaining iterations are pure waste. Stop as soon as it stops shrinking.
6. Two parallel agents, and the second needs the first's output. How do you wire it?
I do not — the premise is the bug. Parallel branches are isolated, so a sibling's output_key is not visible during the block. The interpolation resolves to an empty string, no error is raised, and the model writes something plausible around the hole. It looks like it worked.
If B needs A's output then they are not independent, and putting them in a parallel block is a statement that they are. The correct shape is sequential, or fan-out then join: parallel legs gather independently, and a following stage reads all their outputs, which by then are all in state. If I genuinely need partial overlap, I split it — A alone, then a parallel block containing B and anything else that only needs A.
7. Are “skills” an Anthropic thing, or does Google ADK use them too?
Both, and that is the point. Anthropic originated and published the format, and ADK ships a full implementation — a skills module with the data model, a registry, local and cloud loaders, and a toolset that exposes them to a model. A SKILL.md written to the Claude conventions loads into an ADK agent unmodified.
The detail that settles it is that ADK's own source cites the agentskills.io specification — a vendor-neutral standard, not a competitor's docs. A framework does not reference an external spec for a format it considers someone else's property.
So it belongs in the same family as MCP and A2A: a protocol that started at one vendor and became shared. MCP standardises reaching a tool, A2A reaching another agent, and skills acquiring instructions. In all three cases the value comes from more than one implementation existing.
8. When is something a skill rather than a tool or another agent?
I ask what I am actually adding. A capability the model cannot perform itself — querying a database, sending mail — is a tool, because code has to execute. Knowledge or procedure — a refund policy, a house style, a checklist — is a skill. A separate reasoning context with its own model, tools or authority is an agent.
The tiebreaker I use: imagine pasting the content permanently into the system prompt. If that works but is wasteful, it is a skill and the win is context economy. If it is impossible because code must run, it is a tool. If it would conflict with the instructions already there, it is an agent, because what you need is isolation.
Skills matter because of progressive disclosure. Only names and descriptions go into the prompt up front; bodies load on demand. That is what makes fifty skills affordable, and it is really retrieval applied to instructions rather than documents.
9. When would you build a fully autonomous agent instead of a fixed pipeline?
Rarely, and only when the sequence genuinely cannot be specified in advance — when the task branches on what earlier steps discover. Autonomy is not the advanced option; it is what you fall back to when specification is impossible, and it costs every guarantee that specification provided.
What I give up: predictable cost, bounded latency, and debuggability. A failed run in a fixed pipeline means finding the stage that broke. A failed autonomous run means reconstructing a decision sequence that will never occur again.
Most tasks described as needing autonomy are unspecified pipelines. “Handle customer emails” sounds open-ended and is usually classify, retrieve, draft, check — four known stages, cheaper and far more debuggable as a sequential agent.
10. What bounds an autonomous agent when you cannot predict what it will do?
Four things, in order of leverage. A step cap, set from the 95th percentile of successful runs on my eval set plus headroom, rather than a round number that feels generous. A spend cap, because steps are a poor proxy for cost — one step that reads a long document outweighs twenty that do not.
Then the one that matters most: read-only by default. An autonomous agent with only read tools has a worst case of wasting money and being wrong. Add one side-effecting tool and the worst case becomes unbounded external damage. Every write is a separate, approved, idempotent tool.
And stall detection, because an agent that has stopped progressing still looks busy — it will re-read the same page and rephrase the same query until the budget is gone. One structural detail: I attach specialists as tools, never as sub-agents. A transfer would hand control away permanently and the loop's bounds would stop applying, which defeats the entire arrangement.
11. Run a multi-agent flow over 100,000 rows. What breaks first?
Three things, and I would address all three before starting. First, session reuse: each row needs its own session, or row 2 inherits row 1's history and starts producing summaries of supplier B that mention supplier A. It is a correctness failure with no crash, so at batch scale nobody notices.
Second, concurrency. A bare gather over 100,000 rows tries to call the model 100,000 times at once, hits rate limits in seconds, and the retry storm compounds it. A semaphore bounds in-flight work regardless of batch size.
Third, failure isolation. One row raising must not end the run — return_exceptions=True plus a per-row try turns a fatal error into a recorded one.
I would also do the arithmetic first. That pipeline is up to eight model calls per row, so 800,000 calls; at two seconds each with concurrency of eight that is over fifty hours. That calculation takes a minute and usually changes the design.
12. The batch job dies at hour three. What did you build so that this is survivable?
Results appended per row as they land, rather than collected in memory and written at the end. That single choice is the difference between a rerun that skips 30,000 completed rows and one that repeats three hours of work.
A dead-letter file holding the row, not just the error. Retrying then means filtering the failures, fixing the cause, and resubmitting exactly those rows. A dead-letter file of stack traces tells me what broke and leaves me to reconstruct what to re-run.
A lock around the writes, since concurrent coroutines otherwise interleave and produce lines that are half one JSON object and half another — a file that parses fine until it does not.
And before any of it, a 100-row sample that I read myself. Not the pass rate, the actual text. Systematic problems no validator catches — every summary opening with the same clause, one column consistently misread — are obvious in ten samples and invisible in an aggregate score.
Where this leaves you
You can now build a multi-agent system in ADK and say precisely which of its properties are guaranteed. The loop cannot exceed its bound. The specialist cannot silently keep the conversation, because you chose call-and-return deliberately. A claim without a resolvable citation does not reach the reader. One bad row in eighty thousand is a line in a dead-letter file rather than a failed overnight run.
None of those guarantees came from the model cooperating, and that is the thread running through the chapter. The framework contributes structure — a well-defined place to attach a validator, a cap that holds when the model is most confused, isolation between parallel branches. Everything that determines whether the output can be trusted is ordinary Python you wrote and could carry to another framework tomorrow. The evidence ledger, the tiered validator, the repair loop, the dead-letter queue: all of it is portable, and keeping it that way is what stops a framework choice from becoming a permanent one.
The failures worth remembering are the quiet ones. An escalation that is set and discarded, a sibling branch that reads an empty string, a session reused across rows — none of them raise, none of them log, and all of them produce output that reads correctly. They are found by an invoice, an auditor, or a customer. The defence is not more careful reading; it is measuring the behaviour you are relying on, and then breaking it on purpose to confirm your test noticed. That habit caught a wrong claim in the drafting of this very chapter, which is as good an argument for it as any.
That closes the course. Chapter 1 began with a model that could only produce text, and the arc since then has been about everything required to make that useful and safe: giving it knowledge it does not have, serving it at a cost that works, giving it tools that change the world, and — here — giving it structure that holds when it is wrong. The last part is the one that turns a demonstration into a system.