Chapter 10 · Production serving

Building Agents with Google ADK

Chapter 8 argued that the agent loop is trivial and the harness is everything. This chapter builds that harness in one specific framework — grounded loops that can prove a claim, pipelines that compose four control-flow shapes, agents that choose their own next step, and a batch job that runs a multi-agent flow per row without one bad row taking down the run.

27 sections google-adk 2.8.0 Every snippet executed 12 interview drills Reading time ~3.5 hours

[!] What this chapter assumes, and what it adds

Section 8.25 already surveyed ADK: the agent types, the operational surface, A2A and agent cards, and an honest verdict on when adopting it is worth the coupling. This chapter does not repeat that survey. It assumes you have read it and have decided to build something.

The difference is depth and direction. 8.25 answers should I use this? This chapter answers how do I build the thing, and which of my instincts about it are wrong? Several are: an escalation that looks correct is silently discarded, a sub-agent is stickier than it appears, and a parallel branch cannot see what its sibling just produced. Each of those is demonstrated by running it.

[+] Everything here was executed against ADK 2.8.0

Agent frameworks move fast and their documentation lags, so no API in this chapter is quoted from memory. Every code block was run against a real google-adk 2.8.0 install, every claim about behaviour comes from a measurement, and where the measurement contradicted the intuitive answer, the measurement won and the surprise is called out. The numbers in the figures are the numbers those runs produced.

10.1 What ADK actually gives you

Strip away the branding and ADK offers one idea: control flow between agents should be structure in your code, not an instruction you hope a model follows.

It is worth being concrete about what that buys, because “orchestration framework” is vague enough to sound like it might be doing nothing. Consider a reviewer agent that should try at most three times to fix a draft. Written as a prompt, “attempt this at most three times” is a suggestion; a confused model that has lost track of its own history will happily attempt it nine times, and the failure mode is a bill rather than an error. Written as LoopAgent(max_iterations=3), four attempts is not a thing that can happen. The constraint holds precisely when the model is behaving worst, which is the only time constraints matter.

That is the same argument as 8.17's permission boundaries and 8.21's approval gates, applied to control flow instead of authority. It is also the honest limit of what a framework can do for you: ADK can guarantee how many times something runs and who is allowed to run it. It cannot guarantee that what runs is any good. The parts of this chapter that deal with quality — the validator, the entailment check, the ledger — are code you write, and the framework's role is only to give them a well-defined place to sit.

[def] The five things you will actually use

LlmAgent
A model, an instruction, a toolset. Everything else composes these.
Session state
A dictionary shared across the run. It is how agents pass work to each other, and the reason most pipelines need no glue code at all.
Workflow agents
SequentialAgent, ParallelAgent and LoopAgent — control flow as composition. See 10.16 for their deprecation status, which is genuinely awkward.
Callbacks
Hooks before and after an agent or a tool runs. This is where validation, redaction and budget enforcement live.
Tools
Plain Python functions, other agents wrapped as tools, or whole toolsets such as MCP servers and skills.

10.2 LlmAgent, and what state is for

The smallest useful agent is three fields and one that is easy to overlook.

The smallest useful ADK agentpython
from google.adk.agents import LlmAgent

# An LlmAgent is a model, an instruction and a toolset. Nothing more.
# `output_key` is the important part: whatever this agent returns is
# written to session state under that name, where later agents read it.
drafter = LlmAgent(
    name="drafter",                       # must be unique among siblings
    model="gemini-2.0-flash",
    description="Writes a first draft of a company overview.",
    instruction="Draft a two-sentence overview of the company.",
    output_key="draft",                   # -> state["draft"]
)

output_key is the field that turns a collection of agents into a pipeline. Whatever the agent produces is written into the session's state dictionary under that name, and any later agent can read it by interpolating {draft} into its own instruction. There is no message passing to write and no return values to thread through — state is the bus.

[!] Two fields that look decorative and are not

  • name must be unique among siblings, because it is the identifier used for transfers, for state namespacing, and in every trace you will ever read while debugging. Names like agent1 make a production trace unreadable.
  • description is not documentation. When one agent chooses among several others, the description is what the model reads to decide. An agent with a great instruction and a vague description will be correct whenever it is chosen, and it will rarely be chosen.

The distinction between instruction and description catches people repeatedly, so it is worth stating as a rule: the instruction is read by this agent's model and says how to do the work; the description is read by other agents' models and says when to pick this one. They have different audiences and should not be copies of each other.

10.3 Testing without an API key

Before building anything larger, it is worth solving the problem that otherwise makes agent code untestable: every run costs money, takes seconds, and returns something slightly different. A test suite built on real model calls is slow, flaky, expensive, and cannot reproduce the specific failure you are trying to fix.

ADK only ever calls one method on a model, so satisfying that one method gives you a complete stand-in.

A fake model, so tests need no API keypython
from typing import AsyncGenerator

from google.adk.models import BaseLlm, LlmResponse
from google.genai import types


class FakeLlm(BaseLlm):
    """A model that returns scripted text. No network, no API key.

    Subclassing BaseLlm is the supported extension point: ADK only ever
    calls generate_content_async, so anything satisfying this signature
    is indistinguishable from Gemini to the rest of the framework.
    """

    model: str = "fake"
    replies: list[str] = []

    async def generate_content_async(
        self, llm_request, stream: bool = False
    ) -> AsyncGenerator[LlmResponse, None]:
        text = self.replies.pop(0) if self.replies else "done"
        yield LlmResponse(
            content=types.Content(role="model", parts=[types.Part(text=text)])
        )

[+] Why this is worth doing on day one

A scripted model lets you write the test you actually want: given a model that fabricates a revenue figure on its first attempt and corrects itself on the second, my pipeline must call the model exactly twice and emit the corrected version. That is a precise assertion about your harness, it runs in milliseconds, and it is impossible to write against a live model. Every behavioural claim in this chapter was verified this way — including the ones that turned out to be wrong.

10.4 Running an agent

A runner connects an agent tree to the services it needs — sessions, artifacts, memory. InMemoryRunner wires up in-memory versions of all of them, which is exactly right for tests and for development, and exactly wrong for production, where a restart would erase every conversation.

Driving an agent and reading final statepython
import asyncio

from google.adk.runners import InMemoryRunner
from google.genai import types


async def run_once(agent, prompt: str) -> dict:
    """Drive an agent to completion and return the final session state."""
    runner = InMemoryRunner(agent=agent, app_name="demo")

    # A session is one conversation. Its `state` dict is the channel every
    # agent in the tree reads from and writes to.
    session = await runner.session_service.create_session(
        app_name="demo", user_id="u1"
    )

    # run_async yields every event as it happens: model output, tool calls,
    # state updates. Draining it runs the agent to completion.
    async for _ in runner.run_async(
        user_id="u1",
        session_id=session.id,
        new_message=types.Content(role="user", parts=[types.Part(text=prompt)]),
    ):
        pass

    final = await runner.session_service.get_session(
        app_name="demo", user_id="u1", session_id=session.id
    )
    return dict(final.state)

Two things in that snippet are worth dwelling on. First, run_async is an event stream rather than a function returning an answer: you receive model output, tool calls, and state changes as they happen, which is what makes progressive UI streaming (9.12) and live tracing possible. Draining it in a loop, as here, is simply the batch case.

Second, the final answer is read from session state, not from a return value. This feels indirect at first and pays off immediately: a pipeline of six agents needs no plumbing between them, and every intermediate result is still there afterwards for debugging, which is precisely what you want when the sixth agent produces something strange and you need to know which of the first five caused it.

10.5 Functions as tools

A tool in ADK is a plain Python function. There is no registration step and no schema to write, because the schema is derived from the signature and the docstring.

That convenience has a consequence worth internalising: the docstring is production code. It is not a note for the next developer, it is the text the model reads when deciding whether this function is the right one to call and what to pass it. Section 8.4 made this argument in general terms; ADK makes it literal, because there is no separate schema in which to say it better.

[!] Type hints are not optional here

An unannotated parameter gives the model no information about what to put in it, and the resulting argument will be a plausible-looking string a good deal of the time. A parameter typed order_id: str with a docstring line saying it is a 12-character identifier beginning with ORD- gets the right thing far more often. The general rule from 8.4 applies: make the wrong call hard to express, rather than detecting it afterwards.

10.6 Sub-agents versus AgentTool

This is the most consequential design decision in ADK, and the two options look almost identical when you write them. Both let one agent make use of another. What differs is who is in control afterwards — and the answer is stickier than most people expect.

Transfer or call: the two kinds of delegationpython
from google.adk.agents import LlmAgent
from google.adk.tools import AgentTool

specialist = LlmAgent(
    name="refunds",
    model="gemini-2.0-flash",
    description="Handles refund eligibility and processing.",   # <- routing key
    instruction="Decide refund eligibility from the order record.",
)

# OPTION A -- sub_agents: a TRANSFER, and it is sticky.
# Control moves to the specialist and stays there. The specialist answers
# the next user turn too. Right for "this conversation is now about
# refunds"; wrong if the parent still has work to do afterwards.
router = LlmAgent(
    name="router",
    model="gemini-2.0-flash",
    instruction="Route the customer to the right specialist.",
    sub_agents=[specialist],
)

# OPTION B -- AgentTool: a CALL, and it returns.
# The specialist runs, hands its answer back as a tool result, and the
# parent stays in control. Right when you need to combine several
# specialists, or do anything after the specialist finishes.
supervisor = LlmAgent(
    name="supervisor",
    model="gemini-2.0-flash",
    instruction="Answer the customer, consulting specialists as needed.",
    tools=[AgentTool(agent=specialist)],
)

Reading that code, the natural assumption is that the difference is mostly stylistic: one passes work down the tree, the other calls out sideways, and either way you get an answer back. That assumption is wrong, and it is worth seeing exactly how before it costs you an incident.

Transfer versus call Two panels. With sub_agents, control transfers to the specialist and the specialist also answers the following turn. With AgentTool, the specialist returns a result and the root agent remains in control. sub_agents — transfer AgentTool — call and return root turn 1 specialist transfer turn 2 (user replies): specialist still in control root not consulted root turn 1 specialist result turn 2 (user replies): root still in control specialist idle
Figure 10.1 — The same specialist, wired two ways. Measured, not assumed: with sub_agents the specialist authored both replies; with AgentTool the root authored both.

[def] Transfer is sticky

When a parent transfers to a sub-agent, control does not return at the end of the turn. The sub-agent becomes the active agent for the conversation and answers the next user message too, and the one after that, until something transfers control again.

Driven with a scripted model over two turns, the sub_agents version produced two replies both authored by specialist — the root never spoke again. The AgentTool version produced two replies both authored by root. Same specialist, same prompts, opposite ownership of the conversation.

This matters because the failure it produces is subtle rather than loud. A customer asks about a refund, the router transfers to the refunds specialist, the refund is handled correctly, and everyone is happy. Then the customer asks an unrelated question about delivery times — and the refunds specialist answers it, because it is still holding the conversation. It has no delivery tools and a refund-shaped instruction, so it produces something confident and unhelpful. Nothing errored, and no log line says anything went wrong.

[!] Transfers also cost you prompt caching

A detail ADK itself warns about at runtime, and one that does not appear in most tutorials: every transfer swaps the system instruction and the tool list, which changes the prefix of the request. Prompt caching keys on that prefix, so a transfer-heavy design re-sends the whole prompt uncached after each hop. On a chatty multi-agent system this is a real and recurring cost, not a rounding error. It is configurable — context_cache_config exists for exactly this — but the default is to pay it silently.

10.7 Choosing between them

The decision is not about hierarchy or code aesthetics. It is one question: after this other agent finishes, does the current agent still have work to do?

Which delegation mechanism fits
Situation Use Why
The conversation genuinely changes subject and stays there sub_agents Stickiness is the desired behaviour, not a side effect
You need to combine results from two or more specialists AgentTool A transfer surrenders control, so there is nobody left to combine them
Anything must happen after the specialist finishes — formatting, logging, a follow-up question AgentTool Control returns, so “afterwards” exists
The specialist holds credentials the parent must never use Either, but see 8.17 The boundary is enforced by the toolset, not by the delegation style
You are unsure AgentTool Call-and-return is the less surprising default and is trivially reversible

[retail] A support system that got this wrong

A triage agent routes to returns, delivery and billing as sub-agents. It works beautifully in testing, because every test is a single question. In production, the median conversation contains three questions on two topics, and the second question is answered by whichever specialist happened to handle the first. The fix is not more prompt engineering — it is AgentTool, so triage re-decides on every turn, which is what triage means.

10.8 Why prompts cannot enforce grounding

“Only use the provided sources and do not invent facts” is the most common instruction in production AI systems, and it is not an enforcement mechanism. It is a request, addressed to the component whose defining characteristic is that it sometimes produces confident text with no basis.

The gap between a request and an enforcement is easy to see once stated: if the model ignores the instruction, what in your system notices? For most deployments the honest answer is nothing, until a customer does. That is not a prompt-quality problem and no amount of rewording fixes it — the check has to exist outside the thing being checked.

[!] Provider-side search makes this structurally impossible

If you use a built-in grounded-search tool, the search happens inside the provider and your application receives generated prose that mentions sources. Your code never holds the source documents, so there is no text to compare a claim against. You cannot write the validator even if you want to. Owning retrieval — running the search yourself and keeping what came back — is what makes everything in this section possible, and it is a decision made long before the validator is written.

So the first component is not an agent at all. It is a record of what the run has actually seen.

The evidence ledger: what the run has actually seenpython
from dataclasses import dataclass, field


@dataclass(frozen=True)
class Doc:
    """One retrieved document. `doc_id` is what a claim must cite."""

    doc_id: str
    url: str
    text: str


@dataclass
class EvidenceLedger:
    """Every document the run has actually seen.

    This is the piece that makes grounding enforceable rather than
    aspirational. If the framework performs the search and hands the model
    a summary, your code never holds the source text, so "do not invent
    facts" can only ever be an instruction. Holding the documents yourself
    turns it into an assertion you can run.
    """

    docs: dict[str, Doc] = field(default_factory=dict)

    def add(self, doc: Doc) -> None:
        self.docs[doc.doc_id] = doc

    def resolve(self, doc_id: str) -> Doc | None:
        return self.docs.get(doc_id)

    def corpus(self) -> str:
        return "\n".join(d.text for d in self.docs.values())

The ledger is deliberately dull: a dictionary of documents and a way to resolve an identifier. Its value is entirely in existing at all, because it converts “the model should not invent facts” from a hope into a claim about two pieces of text that your code is holding.

10.9 A three-tier validator

The obvious way to check a draft is to ask a model whether it is supported by the evidence. That works, and if it is the only thing you do it is both slower and more expensive than necessary, because most fabrications can be caught with a regex.

Order the checks by cost. The cheap ones run on everything; the expensive one runs only on what survives.

Tiers 1 and 2: free, deterministic, run on every claimpython
import re

# A number in the draft that never appears in any retrieved document is the
# highest-signal fabrication there is: revenue figures, headcounts, dates.
# Checking it costs one regex and no tokens, so it runs on every claim.
_NUMERIC = re.compile(r"\d[\d,]*\.?\d*")


def normalise(token: str) -> str:
    return token.replace(",", "").rstrip(".")


def validate_claim(claim: dict, ledger) -> list[str]:
    """Return a list of problems. Empty list means the claim is clean.

    Tier 1 -- citation integrity: every cited doc_id must resolve. Catches
             the model inventing a plausible-looking source.
    Tier 2 -- literal grounding: every number in the text must appear in
             at least one document the claim actually cites.
    """
    problems: list[str] = []

    cited = [ledger.resolve(i) for i in claim["doc_ids"]]
    for doc_id, doc in zip(claim["doc_ids"], cited):
        if doc is None:
            problems.append(f"cites unknown document {doc_id}")

    if not claim["doc_ids"]:
        problems.append("claim has no citation")

    supporting = " ".join(d.text for d in cited if d is not None)
    for number in _NUMERIC.findall(claim["text"]):
        if normalise(number) not in normalise(supporting):
            problems.append(f"number {number} appears in no cited document")

    return problems

[def] Why numbers are the highest-signal check

Fabricated prose is hard to detect mechanically, because paraphrase is legitimate: “the company is growing quickly” may be a fair reading of a document that never uses those words. Numbers are different. A revenue figure, a headcount, a date or a percentage either appears in the cited source or it does not, and paraphrase does not apply. Numbers are also the fabrications that do the most damage, because they are the parts a reader trusts most and are least likely to verify. One regex catches the highest-cost error class for no tokens at all.

Two details in that code are there because the naive version is wrong. Normalising away commas means 1,200 in the source matches 1200 in the draft — without it the validator produces constant false positives and the team learns to ignore it, which is worse than not having it. And the comparison is against the text of the cited documents only, not the whole corpus: a claim citing document B is not supported by a number that happens to appear in document D.

Then the expensive tier, for the failure the cheap ones cannot see.

Tier 3: entailment, only for claims that survivedpython
async def entails(claim_text: str, evidence: str, judge) -> bool:
    """Tier 3: does the evidence actually support the prose?

    Only reached for claims that already passed tiers 1 and 2, because it
    is the only tier that costs a model call. A claim can cite a real
    document and quote its numbers correctly and still misrepresent it --
    "revenue grew" when the document says it fell -- and that is precisely
    what this catches and the cheap tiers cannot.
    """
    verdict = await judge(
        "Answer only SUPPORTED or UNSUPPORTED.\n"
        f"Evidence:\n{evidence}\n\nClaim: {claim_text}"
    )
    return verdict.strip().upper().startswith("SUPPORTED")


async def validate_all(claims: list[dict], ledger, judge) -> dict:
    """Run the cheap tiers on everything, the expensive tier on survivors."""
    report: dict[str, list[str]] = {}
    for claim in claims:
        problems = validate_claim(claim, ledger)
        if not problems:
            evidence = " ".join(
                ledger.resolve(i).text for i in claim["doc_ids"]
            )
            if not await entails(claim["text"], evidence, judge):
                problems = ["evidence does not support this claim"]
        if problems:
            report[claim["text"]] = problems
    return report

[+] What each tier catches that the others cannot

  • Tier 1, citation integrity. The model invented a source, or cited nothing at all. Free.
  • Tier 2, literal grounding. The source is real but the number is not. Free.
  • Tier 3, entailment. Every citation resolves and every number is real, and the sentence still misrepresents the source — “revenue grew to $1.2 billion” drawn from a document reporting that revenue fell to $1.2 billion. Both cheap tiers pass this happily. Only a reader — human or model — catches it.

That last example is not hypothetical; it is the test case in the executed version of this code. The cheap tiers return no problems for it, and the judge rejects it. The same test also asserts the ordering works in the other direction: a claim with a fabricated number never reaches the judge at all, so the expensive tier is not spent on something already known to be broken.

10.10 The escalation that gets discarded

Here is the bug that motivates writing this section at all. It is invisible in review, it does not error, the output it produces is correct, and it multiplies your model spend by the loop bound.

A validator lives naturally in an after_agent_callback: the drafter runs, the callback inspects what it produced, and if everything checks out it asks the loop to stop. Asking the loop to stop means setting escalate. Both versions below do that, and only one works.

The same validator, broken and workingpython
from google.adk.agents.callback_context import CallbackContext


def validator_BROKEN(callback_context: CallbackContext):
    """Looks correct. Does not stop the loop."""
    callback_context.actions.escalate = True
    return None


def validator_WORKING(callback_context: CallbackContext):
    """Identical intent, but the escalation actually survives.

    ADK only turns the callback's actions into a real event if the callback
    returns content or leaves a state delta. With neither, the escalate flag
    is constructed and then dropped on the floor, and the loop runs to
    max_iterations at full token cost -- while still producing correct
    output, which is what makes it so easy to miss.
    """
    callback_context.actions.escalate = True
    callback_context.state["validation_passed"] = True   # <- the load-bearing line
    return None
Why an escalation is sometimes ignored A branch. If the callback returns content or writes state, the escalation becomes an event and the loop stops after 1 pass. If it does neither, the flag is discarded and the loop runs 5 times. after_agent_callback sets escalate = True did the callback return content or leave a state delta? no flag discarded no event is created loop runs all 5 yes event carries it loop sees escalate stops after 1 same intent, 5× the model calls output still looks correct, which is why it survives review
Figure 10.2 — Measured on ADK 2.8.0: setting escalate alone ran 5 iterations; adding a state write stopped it after 1.

[!] Measured, not theorised

With max_iterations=5 and a model whose very first draft is clean: setting escalate alone ran the drafter 5 times. Adding a single state write stopped it after 1. Identical intent, identical output, 5.0× the model calls.

The cause is in ADK's own handling of the callback. It only constructs an event carrying the callback's actions if the callback returned content or left a state delta. With neither, the escalate flag is set on an object that is then discarded, so the loop never learns anything happened. The reason this survives review is that the output is still correct — the extra iterations regenerate an already-good draft, the final answer looks right, and the only symptom is the invoice.

[def] A regression test must fail when you break it

This bug is a good argument for mutation testing, and it earned that reputation honestly: an earlier version of this material described the rule as “the callback must return content,” which is only half true. That error was caught by deliberately reintroducing the bug and discovering the test still passed — it was accidentally protected by an unrelated state write elsewhere. If you write a test for this, break the code on purpose and confirm the test goes red. A test that passes either way is not a test.

There is also a way to avoid the trap entirely. ADK ships the escape hatch as a tool.

The idiomatic exit: escalate from a toolpython
from google.adk.tools import exit_loop

# ADK ships the escape hatch as a tool. Given to an LlmAgent inside a
# LoopAgent, the model calls it when its own instruction says the work is
# done, and the loop stops. Because a tool call is already a real event,
# the escalation is never discarded -- the failure mode above cannot occur.
reviewer = LlmAgent(
    name="reviewer",
    model="gemini-2.0-flash",
    instruction=(
        "Review the draft in {draft}. If every claim is supported, call "
        "exit_loop. Otherwise reply with the specific problems to fix."
    ),
    tools=[exit_loop],
)

Because a tool call is already a real event, an escalation raised from tool context cannot be discarded — the failure mode above simply does not apply. It also moves the stop decision into the model's reasoning, where it is visible in traces as an explicit action rather than inferred from a callback's side effects. Prefer this when the stopping criterion is a judgement call. Keep the callback approach when the criterion is deterministic, as it is for the validator, since you do not want to ask a model whether the regex matched.

10.11 The repair loop

With a validator and a working escalation, the loop writes itself — but one detail separates a repair loop from an expensive retry.

The repair loop, with an escalation that survivespython
from google.adk.agents import LlmAgent, LoopAgent
from google.adk.agents.callback_context import CallbackContext


def make_grounding_gate(ledger):
    """after_agent_callback that stops the loop only when the draft is clean.

    Note both halves of the escape: `escalate` states the intent, and the
    state write is what makes ADK materialise an event carrying it. Setting
    only the flag leaves the loop running to max_iterations.
    """

    def gate(callback_context: CallbackContext):
        draft = callback_context.state.get("draft", "")
        claims = parse_claims(draft)
        problems = {
            c["text"]: p
            for c in claims
            if (p := validate_claim(c, ledger))
        }

        if not problems:
            callback_context.actions.escalate = True
            callback_context.state["grounded"] = True     # makes escalate stick
            return None

        # Feed the specific failures back so the next pass is a repair
        # rather than a reroll. The drafter reads {problems} in its prompt.
        callback_context.state["problems"] = problems
        return None

    return gate


def build_grounded_loop(model, ledger, max_iterations: int = 3) -> LoopAgent:
    drafter = LlmAgent(
        name="drafter",
        model=model,
        instruction=(
            "Write the overview using only the supplied evidence. Cite a "
            "doc_id for every claim. Problems to fix: {problems?}"
        ),
        output_key="draft",
        after_agent_callback=make_grounding_gate(ledger),
    )
    # max_iterations is a structural bound, not a request. The loop cannot
    # exceed it however confused the model becomes.
    return LoopAgent(name="grounded", sub_agents=[drafter],
                     max_iterations=max_iterations)

[+] Feed back the specific failure, not the fact of failure

Writing the problems into state and interpolating them into the drafter's instruction is what makes the second attempt a repair. Simply re-running the drafter is a reroll: the same prompt, the same evidence, and a fresh sample from the same distribution that just produced a fabrication. Telling it exactly which number was unsupported changes the input, and therefore changes the odds. This is the same principle as 8.7's structured tool errors — an error message is only useful if it says what to do differently.

The executed version of this loop asserts three behaviours: a clean first draft stops after one pass; a draft with a fabricated number is caught and the corrected second draft ends the loop after exactly two; and a model that never gets it right terminates at max_iterations rather than running forever. That third case is the one worth testing deliberately, because it is the one that only shows up in production, on the input nobody thought about.

10.12 Stall detection and quarantine

A bounded loop cannot run forever, but it can still waste its entire budget usefully doing nothing. Two refinements are worth the small amount of code they cost.

Stall detection. If the set of problems after this pass is identical to the set before it, the model is not converging — it is producing the same failure with different wording. Three iterations of that is three times the cost of one, with the same outcome. Compare the problem set between passes and stop early when it stops shrinking; you have learned everything the loop is going to teach you.

Quarantine. When the loop exhausts its budget with problems outstanding, the question is what to publish. Publishing the draft with its unsupported claims intact is the worst option and, unfortunately, the default if nobody decides otherwise. The better move is to drop the specific claims that failed and emit the rest, marked as incomplete — the section that says “revenue: not found” is far more useful than the one that invents a number, and it is the behaviour 5.31's abstention discussion argues for.

[!] Make quarantine monotonic

Once a claim has failed validation, it should stay out. A subsequent pass that reproduces the same unsupported sentence must not be able to reintroduce it by happening to phrase it in a way the validator does not catch — otherwise the loop's last iteration can undo the work of all the earlier ones. Track rejected claims across iterations, not within one.

10.13 Sequential, parallel, loop

Three control-flow shapes, each corresponding to a property of the work rather than a preference about code. Choosing between them is a question about your task, not about the framework.

Which shape the work implies
Shape Use when The property being exploited
SequentialAgent Each stage needs the previous stage's output Dependency. There is no choice, and no latency to win back.
ParallelAgent Sub-tasks are genuinely independent Independence. Latency becomes the slowest leg rather than the sum.
LoopAgent Quality is reached by iteration, not in one pass Convergence — with a structural cap, since convergence is not guaranteed.

The measurement is worth stating plainly, because “parallel” in an async framework is not always as parallel as advertised. Three legs of 0.4s each, run through a ParallelAgent, completed in 0.41s against 1.2s if run one after another — a 3.0× speedup, which is the whole of the theoretical maximum. The branches genuinely overlap.

[!] A measurement I got wrong first time

The first version of that experiment reported 2.02s for two 0.5s legs and I briefly concluded ADK was running them serially. It was not — the harness was measuring setup it should not have been. Re-running it with per-call timestamps showed both branches starting at the same instant and ending at the same instant. The lesson is the one from chapter 19: a single wall-clock number is a weak measurement, and the fix is to instrument the thing you actually care about rather than time the whole program.

10.14 What parallel branches can see

Parallel sub-agents run in isolated branches, and the consequence is the thing people get wrong: a branch cannot read what a sibling produced during the same parallel block. If one leg writes findings_people and another leg's instruction interpolates {findings_people}, that placeholder resolves to nothing.

[def] Isolated during, merged after

Every branch's writes do land in session state, and after the parallel block completes they are all visible to whatever runs next. So the pattern that works is fan-out then join: parallel legs gather independently, and a following sequential stage reads all of their outputs. The pattern that silently produces empty strings is one leg trying to build on another leg's work. If leg B needs leg A's output, they are not independent, and they should not be in the same parallel block.

This is worth checking against your intuition, because the failure is quiet. There is no error and no warning; the interpolation resolves to an empty string, the model writes something plausible around the gap, and the result reads fine. It is the same class of problem as the discarded escalation — correct-looking output hiding a structural mistake.

10.15 A full composed pipeline

Now the scenario that combines everything: a research report that plans its own work, gathers evidence on several fronts at once, drafts and re-drafts under validation, and assembles a final document. Four agents, three control-flow shapes, one validator.

A composed pipeline A sequential pipeline: planner, then a parallel agent with three isolated research branches, then a bounded loop containing a drafter and a validator gate, then an assembler. SequentialAgent planner LlmAgent ParallelAgent financials isolated branch people isolated branch risks isolated branch LoopAgent drafter LlmAgent gate validator max_iterations = 3 assembler LlmAgent
Figure 10.3 — Each shape is chosen for a property: sequential for dependency, parallel for independence, loop for convergence. Nesting is what makes them useful.
Sequential, parallel and loop in one pipelinepython
from google.adk.agents import LlmAgent, LoopAgent, ParallelAgent, SequentialAgent


def build_pipeline(model, ledger) -> SequentialAgent:
    """Plan -> (research x N in parallel) -> draft/critique loop -> assemble.

    Each shape is chosen for a property, not for variety:
      Sequential -- later stages need earlier stages' output.
      Parallel   -- the research legs are independent, so latency is one
                    leg rather than the sum. Branches are isolated: a
                    sibling's output_key is NOT visible mid-run.
      Loop       -- quality is a fixed point, not a single pass, and the
                    iteration cap is structural.
    """
    planner = LlmAgent(
        name="planner", model=model, output_key="plan",
        instruction="Break the request into research questions.",
    )

    legs = [
        LlmAgent(name=f"research_{topic}", model=model,
                 output_key=f"findings_{topic}",
                 instruction=f"Research {topic} for: {{plan}}")
        for topic in ("financials", "people", "risks")
    ]
    research = ParallelAgent(name="research", sub_agents=legs)

    drafter = LlmAgent(
        name="drafter", model=model, output_key="draft",
        instruction=(
            "Draft from {findings_financials}, {findings_people}, "
            "{findings_risks}. Fix any problems in {problems?}."
        ),
        after_agent_callback=make_grounding_gate(ledger),
    )
    refine = LoopAgent(name="refine", sub_agents=[drafter], max_iterations=3)

    assembler = LlmAgent(
        name="assembler", model=model, output_key="report",
        instruction="Format {draft} as the final report.",
    )

    return SequentialAgent(
        name="pipeline", sub_agents=[planner, research, refine, assembler]
    )

[+] Read the structure, not the prompts

The valuable property of this code is that you can determine what will happen without reading a single instruction string. Planning happens once. Research happens three ways at once. Drafting happens between one and three times. Assembly happens last, and exactly once. None of that is inferable from a prompt-driven design, where the same behaviour would be a paragraph of English and a hope. This is what 8.25 meant by structural orchestration, and it is the strongest argument for a framework like this one.

Note also what the shapes cost. The parallel block turns three sequential model calls into one wall-clock leg, which is the difference between a report that takes ninety seconds and one that takes half a minute. The loop's bound means a pathological input costs at most three drafts instead of an unbounded number. Both are properties of the structure, and neither depends on the model cooperating.

[retail] Where the seams go in a real system

A supplier-risk report at a retailer looks exactly like this: plan the questions, then hit financial filings, news, and internal incident history simultaneously — three independent sources, no reason to serialise them — then draft under a validator that refuses any risk claim without a resolvable citation, because this document informs a purchasing decision and an invented risk is as damaging as a missed one. The assembler exists so the output format is decided in one place rather than being something each drafting pass argues with itself about.

10.16 The deprecation problem

An honest note, because this will confuse anyone reading the source. In ADK 2.8.0 all three workflow agents — SequentialAgent, ParallelAgent and LoopAgent — are marked deprecated in favour of a newer graph-based Workflow API. Importing them emits a DeprecationWarning.

[!] Deprecated, and still the only option

The deprecation notice carries its own caveat: “Workflow cannot yet be used as an LlmAgent sub-agent.” So if you need a loop or a parallel block nested inside an agent — which is what every example in this chapter does — the deprecated classes remain the only way to express it. Workflow is also not exported from google.adk.agents yet; it lives in a private module.

The practical response is not to avoid the deprecated classes, which would mean not building the thing. It is to confine them. Keep every construction of a workflow agent inside one small module that exposes your own pipeline-building functions, as build_pipeline does above. Then the migration, when Workflow is ready, is one file rather than a search across the codebase — and your business logic, which does not care how the loop is implemented, never mentions the deprecated names at all.

This is a specific instance of a general habit worth having with fast-moving frameworks: the parts of a young library most likely to be rewritten are its orchestration primitives, and the parts least likely are plain functions and data. Keep your value in the latter. It is also the same advice 8.25 gives about ADK as a whole — use the operational surface, but keep your tool implementations and business rules in ordinary Python that would survive changing frameworks entirely.

10.17 Is “skills” a Claude word?

It is a reasonable suspicion. Agent skills arrived with a great deal of Anthropic branding, and vendor-specific vocabulary dressed up as an industry standard is common enough that scepticism is the right default. In this case the scepticism is misplaced, and the way to settle it is to look in the package rather than at the marketing.

Google ADK 2.8.0 ships a google.adk.skills module. It is not a compatibility shim or a thin adapter — it contains the data model (Skill, Frontmatter, Resources, Script), a registry, loaders for both local directories and cloud storage, and a SkillToolset that exposes skills to a model. Loading a SKILL.md file written to the Claude conventions into an ADK agent works without modification.

[+] The decisive evidence is in the source

ADK's own docstring for the allowed_tools field points the reader at agentskills.io/specification — a vendor-neutral specification, not Anthropic's documentation. A framework does not cite an external standard for a format it considers proprietary to a competitor. Skills are a cross-vendor convention that Anthropic originated and published; ADK implements it.

So the accurate framing is the same one 8.9 used for MCP and 8.25 used for A2A. All three are protocols that began at one vendor and became shared, and their value comes precisely from being implemented in more than one place. Skills are the third member of that family: MCP standardises how an agent reaches a tool, A2A how it reaches another agent, and skills how it acquires instructions.

10.18 Writing and loading a skill

A skill is a folder, not a Python object. That is the entire idea, and it is what makes skills portable between frameworks in a way that a class hierarchy never could be.

The layout of a skilltree
skills/
  refund-policy/
    SKILL.md          # required: YAML frontmatter + markdown instructions
    references/       # optional: deeper documentation, loaded on demand
    assets/           # optional: templates and other resources
    scripts/          # optional: executables the agent may run

The frontmatter carries the metadata; the body carries the instructions.

skills/refund-policy/SKILL.mdmarkdown
---
name: refund-policy
description: Apply the refund policy. Use when a customer asks for a
  refund, a return, or an exchange.
license: Apache-2.0
metadata:
  adk_inject_state: true
---

# Refund policy

The customer's loyalty tier is {tier?}.

1. Within 30 days of delivery, with proof of purchase, issue a full refund.
2. Between 30 and 90 days, issue store credit.
3. After 90 days, no refund. Offer a repair booking instead.

Loading it is two lines.

Loading skills from disk into an agentpython
from google.adk.agents import LlmAgent
from google.adk.skills import load_skills_from_dir
from google.adk.tools.skill_toolset import SkillToolset

# A skill is a folder on disk, not a Python object:
#     skills/refund-policy/SKILL.md        <- YAML frontmatter + instructions
#     skills/refund-policy/references/     <- optional deeper docs
#     skills/refund-policy/scripts/        <- optional executables
#
# Only each skill's name and description are put in the prompt up front.
# The body is loaded on demand, when the model decides the skill is
# relevant -- which is the whole point: you can ship fifty skills without
# paying fifty skills' worth of context on every turn.
skills = load_skills_from_dir(SKILLS_DIR)

agent = LlmAgent(
    name="support",
    model="gemini-2.0-flash",
    instruction="Help the customer. Use a skill when one applies.",
    tools=[SkillToolset(skills=skills)],
)

[def] Progressive disclosure is the whole point

Only each skill's name and description are placed in the prompt up front. The body is fetched only when the model decides the skill is relevant and calls load_skill. That is what makes fifty skills affordable: you pay fifty short descriptions per turn instead of fifty full documents. The executed version of the snippet above asserts this directly — it confirms the phrase “store credit” is present in the skill body and absent from the block sent up front.

Two ADK-specific extensions live in metadata, and both are worth knowing. adk_inject_state: true enables {variable} interpolation in the skill body from session state — so the refund policy above can address the customer's actual loyalty tier, with {tier?} resolving to an empty string rather than raising when the key is missing. adk_additional_tools lets a skill declare tools that become available when it is loaded.

[!] The description is a routing decision, again

As with an agent's description in 10.2, this is the field that determines whether the skill is ever used. “Refund policy” is a label; “Apply the refund policy. Use when a customer asks for a refund, a return, or an exchange” is a routing instruction that names the trigger conditions. ADK enforces a 1024-character limit here, which is generous — the constraint is not length, it is remembering that this text has a job to do.

10.19 Skills versus tools versus agents

Three mechanisms for extending what an agent can do, and a reliable way to choose: what are you actually adding?

Choosing an extension mechanism
You are adding Use Because
A capability — something the model cannot do itself, like querying a database or sending an email A tool Code executes; the model only decides when
Knowledge or procedure — a policy, a house style, a multi-step checklist A skill It is instructions, and it is too long to keep in the prompt permanently
A separate reasoning context — different model, different tools, different authority An agent You need isolation, not just extra text or one function

[+] A useful test when two of them seem to fit

Ask what would happen if you pasted the content into the system prompt permanently. If the answer is “that works, it is just wasteful” — it is a skill, and the win is context economy. If the answer is “that is impossible, it needs to run code” — it is a tool. If the answer is “that would conflict with the instructions already there” — it is an agent, because what you actually need is a separate context, and this is 8.17's argument restated.

It is worth noting that skills partly close the gap that made 8.14's context engineering so difficult. The old dilemma was between a bloated system prompt that covered every eventuality and a lean one that failed on unusual cases. Skills reframe it: keep the system prompt lean, keep the long-tail procedures on disk, and let the model fetch what the current situation calls for. That is retrieval — applied to instructions rather than documents, which is a genuinely good idea and slightly overdue.

10.20 Fixed pipelines versus autonomy

Everything so far has had its control flow decided in advance. 10.15's pipeline always plans, then researches, then drafts, then assembles. The model chooses the words; the code chooses the sequence. A fully autonomous agent inverts that: the model chooses the sequence too.

The temptation is to treat autonomy as the more advanced option and fixed pipelines as the beginner's version. That gets the trade backwards. Autonomy is what you resort to when you cannot specify the sequence, and it costs you every guarantee that came from specifying it.

What you gain and what you give up
Property Fixed pipeline Autonomous agent
Cost per run Predictable within a narrow range Varies by an order of magnitude between inputs
Latency Bounded by the structure Bounded only by the step cap
Debugging a bad run Find the stage that failed Reconstruct a sequence of decisions that will not recur
Handles tasks whose shape is unknown in advance No — it does what it was built to do Yes, and this is the entire reason to accept the above
Safe with side-effecting tools Auditable up front Requires every control in 8.20–8.22, without exception

[!] The honest default

Most tasks presented as needing an autonomous agent are a fixed pipeline that nobody has sat down and specified. “Handle customer emails” sounds open-ended and is usually classify, retrieve, draft, check — four stages, known in advance, cheaper and more debuggable as a SequentialAgent. Reach for autonomy when the task genuinely branches on what earlier steps discover, and 17.22's warning about complexity you cannot measure applies directly.

10.21 Building an autonomous agent

The construction is simpler than the fixed pipeline, which is itself a useful signal about where the difficulty has moved.

A fully autonomous agent, with structural boundspython
from google.adk.agents import LlmAgent, LoopAgent
from google.adk.tools import AgentTool, exit_loop


def build_autonomous(model, tools, specialists, max_steps: int = 12) -> LoopAgent:
    """A fully dynamic agent: it decides its own next step each iteration.

    The difference from the composed pipeline is where control flow lives.
    There, the sequence was fixed in code and the model filled in content.
    Here the model chooses the sequence, and the *bounds* are what live in
    code. That trade buys flexibility on tasks whose shape is not known in
    advance, and it costs you the ability to predict what will run.

    Note what remains structural even at full autonomy:
      - max_steps caps the loop no matter what the model believes;
      - specialists are AgentTools, so control always returns here;
      - exit_loop is the only way to finish early, and it is explicit.
    """
    worker = LlmAgent(
        name="worker",
        model=model,
        instruction=(
            "Goal: {goal}\n"
            "Progress so far: {scratchpad?}\n"
            "Take ONE step towards the goal. Append what you learned to the "
            "scratchpad. When the goal is fully met, call exit_loop."
        ),
        tools=[*tools, exit_loop, *(AgentTool(agent=s) for s in specialists)],
        output_key="scratchpad",
    )
    return LoopAgent(name="autonomous", sub_agents=[worker],
                     max_iterations=max_steps)

One agent, in a loop, with a scratchpad. Each iteration it reads the goal and what it has learned, takes one step, and appends the result. It stops when it decides the goal is met and calls exit_loop, or when the step cap is reached. This is 8.3's agent loop and 8.11's planning discussion, expressed in ADK's primitives.

[+] The specialists are tools, deliberately

Note that specialists are wrapped as AgentTool rather than attached as sub-agents. In an autonomous loop this is not a stylistic choice. A transfer would hand control away permanently, and the loop — along with its step cap and its scratchpad — would simply stop being in charge. Wrapping them as tools means the loop always regains control, so its bounds continue to apply. Autonomy is exactly the setting where the sticky-transfer behaviour from 10.6 does the most damage.

10.22 Bounding what you cannot predict

Since you can no longer say what an autonomous agent will do, everything rests on what it cannot do. Four bounds, in decreasing order of how much they save you.

A step cap. max_iterations is the difference between a bad run costing a few dollars and a bad run costing until someone notices. Set it from a measurement — the 95th percentile of successful runs on your evaluation set, plus headroom — rather than from a round number that feels generous.

A spend cap. Steps are a poor proxy for cost, because one step that reads a 200-page document costs more than twenty that do not. Track tokens in a callback and stop on the budget, as 8.22 argues. An agent that has spent its allowance should stop even if it has steps remaining.

Read-only by default. The single highest-leverage control. An autonomous agent with only read tools has a worst case of wasting money and returning something wrong. Add one side-effecting tool and the worst case becomes unbounded external damage. Every write should be a separate, approved, idempotent tool — 8.5 and 8.21 in full, applied without exception.

Stall detection. Same idea as 10.12, and more important here. An autonomous agent that has stopped making progress will keep looking busy: it will re-read the same page, rephrase the same query, and consume its entire budget confidently. Compare the scratchpad between iterations and stop when it stops growing in substance.

Per-agent timeout and retry, built inpython
from google.adk.agents import LlmAgent
from google.adk.workflow._retry_config import RetryConfig

# Per-agent retry and timeout are built in -- there is no need to wrap
# agents in your own retry decorator, and doing so tends to multiply with
# this one rather than replace it.
researcher = LlmAgent(
    name="researcher",
    model="gemini-2.0-flash",
    instruction="Research the question.",
    timeout=60.0,                       # seconds, per agent run
    retry_config=RetryConfig(
        max_attempts=3,
        initial_delay=1.0,
        backoff_factor=2.0,             # 1s, 2s, 4s
        jitter=0.3,                     # avoids a synchronised retry storm
    ),
)

[def] Use the built-in retry, not your own

timeout and retry_config are fields on every agent, so transient failures do not need a decorator you wrote. This matters more than it sounds: a hand-rolled retry wrapped around an agent that is already retrying multiplies rather than replaces, and three attempts times three attempts is nine model calls for one logical step. Jitter is worth setting for the reason 16.14 gives — without it, a batch of agents that fail together will retry together and reproduce the overload that caused the failure.

10.23 One flow per row

A different shape of problem: not one conversation with a user, but a hundred thousand rows in a table, each of which must trigger its own multi-agent flow. Enrich every supplier record. Classify every support ticket from last quarter. Draft a summary for every contract. The agent design is the same; everything around it changes.

The first decision is the one that determines whether the rest works, and it is easy to get wrong in a way that only shows up at scale: each row gets its own session.

Batch fan-out Rows pass through a semaphore that bounds concurrency, each into its own session and agent tree. Successful rows append to a results file; a failed row is written to a dead-letter file with enough detail to resubmit it. one row → one session → one agent tree rows CSV / table 100,000 Semaphore concurrency = 8 bounds in-flight model calls, not total rows session r1 isolated state session r2 isolated state session r3 isolated state raised — recorded results.jsonl appended per row dead-letter row + error, retryable
Figure 10.4 — Verified: one row raising an exception left the other rows unaffected, and the semaphore held peak concurrency at exactly its limit.
One multi-agent flow per row, safelypython
import asyncio

from google.adk.runners import InMemoryRunner
from google.genai import types


async def process_row(row: dict, build_agent, limit: asyncio.Semaphore) -> dict:
    """Run the full multi-agent flow for exactly one row.

    Three properties matter here, and each is one line:
      - a session PER ROW, so rows cannot see each other's state;
      - a semaphore, so 100k rows do not open 100k concurrent model calls;
      - a try/except returning a value, so one bad row is a recorded
        failure rather than a dead batch.
    """
    async with limit:
        runner = InMemoryRunner(agent=build_agent(), app_name="batch")
        session = await runner.session_service.create_session(
            app_name="batch",
            user_id=row["id"],              # isolation key
            state={"row": row},             # seed input, no prompt stuffing
        )
        try:
            async for _ in runner.run_async(
                user_id=row["id"],
                session_id=session.id,
                new_message=types.Content(
                    role="user", parts=[types.Part(text=row["text"])]
                ),
            ):
                pass
        except Exception as exc:            # noqa: BLE001 -- deliberate
            return {"id": row["id"], "status": "failed", "error": str(exc)}

        final = await runner.session_service.get_session(
            app_name="batch", user_id=row["id"], session_id=session.id
        )
        return {"id": row["id"], "status": "ok", "state": dict(final.state)}


async def run_batch(rows, build_agent, concurrency: int = 8) -> list[dict]:
    limit = asyncio.Semaphore(concurrency)
    # return_exceptions=True is the difference between "one row raised" and
    # "the batch died on row 40,000 after four hours".
    return await asyncio.gather(
        *(process_row(r, build_agent, limit) for r in rows),
        return_exceptions=True,
    )

[!] Reusing one session across rows is the classic bug

It is tempting, since creating a session per row feels wasteful. What actually happens is that row 2 inherits row 1's conversation history and state. The model, quite reasonably, treats it as context — and starts producing summaries of supplier B that mention supplier A. Because it is a correctness failure rather than a crash, it is discovered by whoever reads the output, which at batch scale is often nobody. A session is cheap; contamination is not.

Note also how the row's data enters: as session state, not by formatting it into the prompt. This keeps structured input structured, avoids an entire class of injection problems from row content that happens to contain instruction-like text (15.7), and means any agent in the tree can read the original row rather than the model's paraphrase of it.

10.24 Concurrency, failure, and cost

Three properties, each worth one line of code, and each catastrophic to omit.

Bound the concurrency. asyncio.gather over 100,000 rows creates 100,000 coroutines that all try to call the model at once. You will hit rate limits within seconds, and the retry storm that follows will make it worse. A semaphore bounds in-flight work regardless of batch size. Verified: with a limit of 2 and six rows, peak concurrency was exactly 2.

Isolate the failures. One row that raises must not end the run. return_exceptions=True plus a per-row try turns a fatal error into a recorded one. Verified: with six rows and one deliberately raising, five completed and the failure was captured with its row identifier intact.

Know the cost before you start. A multi-agent flow with a plan stage, three parallel research legs and a loop that may run three times is not one model call per row. Count the calls in the worst case, multiply by the row count, and price it before running the job rather than after.

[def] The arithmetic that ends most batch plans

10.15's pipeline is 1 planner + 3 research + up to 3 drafts + 1 assembler = up to 8 model calls per row. Across 100,000 rows that is 800,000 calls. At two seconds each with a concurrency of 8, that is roughly 55 hours of wall-clock time. This calculation takes a minute and routinely changes the design — usually by finding rows that do not need the full pipeline, which is the next point.

[+] Filter before the agent, not inside it

The cheapest model call is the one that does not happen. If 60% of rows can be resolved by a rule, a lookup, or a classifier costing a thousandth as much, run that first and send only the remainder to the agent. Using a multi-agent pipeline to decide that a row is uninteresting is the most expensive possible way to reach that conclusion, and it is a common design because the agent is the new tool and everything looks like a nail.

10.25 Checkpointing and dead letters

A batch job long enough to be worth parallelising is long enough to be interrupted. The difference between a four-hour job that can resume and one that cannot is a few lines.

Streaming results and a dead-letter queuepython
import asyncio
import json


async def run_batch_streaming(rows, build_agent, out_path, dead_path,
                              concurrency: int = 8) -> dict:
    """Write each result as it lands, so a crash at row 90,000 is not fatal.

    Collecting into a list and writing at the end means an OOM or a killed
    pod loses the whole run. Appending per row means a rerun skips what is
    already done. For a long batch this is the difference between a retry
    costing minutes and costing the entire job again.
    """
    limit = asyncio.Semaphore(concurrency)
    tally = {"ok": 0, "failed": 0}
    lock = asyncio.Lock()

    with open(out_path, "a") as out, open(dead_path, "a") as dead:

        async def one(row):
            result = await process_row(row, build_agent, limit)
            async with lock:                      # one writer at a time
                if result["status"] == "ok":
                    out.write(json.dumps(result) + "\n")
                    tally["ok"] += 1
                else:
                    # The dead-letter file is the retry queue: it holds the
                    # row itself, not just the error, so a rerun is trivial.
                    dead.write(json.dumps({"row": row, **result}) + "\n")
                    tally["failed"] += 1

        await asyncio.gather(*(one(r) for r in rows))

    return tally

[+] Two files, two purposes

  • Results, appended per row. Collecting into a list and writing at the end means a crash at row 90,000 loses everything. Appending as you go means a rerun skips what is already done. At batch scale this is the difference between a retry costing minutes and costing the whole job.
  • A dead-letter file, holding the row itself. Not just the error — the input. That makes the retry trivial: filter the rows that failed, fix the cause, and resubmit exactly those. A dead-letter file containing only stack traces tells you what broke and leaves you to reconstruct what to re-run.

The single lock around the writes is doing real work, and it is the kind of detail that is invisible until it is not. Without it, concurrent coroutines interleave their writes and produce lines that are half one JSON object and half another — a file that looks fine until something tries to parse it, which is usually the following morning.

[retail] A supplier-enrichment job, end to end

80,000 suppliers, each needing the 10.15 pipeline. The shape that works: a cheap pre-filter drops the 40% with nothing to enrich; a semaphore of 8 keeps the model within its rate limit; each surviving row gets its own session seeded with the supplier record; results append to JSONL as they land; failures go to a dead-letter file with the row attached. When it dies at hour three because a credential expired, the rerun skips 30,000 completed rows and costs twenty minutes instead of three hours. Every one of those decisions is boring, and together they are the difference between a job that ships and a job that gets abandoned after the second failed overnight run.

[!] Sample before you commit

Run 100 rows and read the output yourself before launching 80,000. Not the pass rate — the actual text. Systematic problems that no validator catches, like every summary opening with the same clause or the model consistently misreading one column, are obvious in ten samples and invisible in an aggregate score. This is 8.26's evaluation argument at batch scale, and the sampling costs a rounding error of the full run.

10.26 Key takeaways

The framework gives you structure. Everything that determines whether the output can be trusted is code you write and place inside that structure.

What to remember from this chapter
Decision The rule Why
Sub-agent or tool Default to AgentTool Transfer is sticky — the specialist keeps answering later turns
Stopping a loop from a callback escalate and a state write, or use exit_loop The flag alone is discarded; measured at 5.0× the model calls
Enforcing grounding Own retrieval, keep a ledger, validate in tiers A prompt cannot check itself; provider-side search makes checking impossible
Validator ordering Citations, then numbers, then entailment Two of the three are free; only spend tokens on what survives them
Parallel branches Fan out, then join — never chain within a block Siblings cannot see each other's output mid-run, and fail silently
Workflow agents Use them, but confine them to one module Deprecated, yet still the only way to nest under an LlmAgent
Skills, tools or agents Instructions, capability, or isolation — pick by what you are adding Each solves a different problem; the overlap is only apparent
Autonomy Only when the sequence genuinely cannot be specified You trade every cost, latency and debugging guarantee for flexibility
Batch Session per row, bounded concurrency, dead letters Contamination, rate limits and lost work are the three failure modes

[+] The pattern behind the three worst bugs here

The discarded escalation, the sibling that reads an empty string, and the reused batch session share a signature: the output still looks correct. Nothing raises, nothing logs a warning, and the result passes a glance. That is what makes them expensive — they are found by an invoice, an auditor, or a customer rather than by a test. When working with a framework whose whole job is to hide control flow from you, the failures worth fearing are the quiet ones, and the only reliable defence is to assert on behaviour you have actually measured.

[def] What is genuinely portable from this chapter

Almost all of it. The evidence ledger, the tiered validator, the repair loop with specific feedback, stall detection, monotonic quarantine, session-per-row, bounded concurrency, dead-letter queues — none of these are ADK ideas. They are the answers to problems that any agent framework leaves for you, and they transfer intact to LangGraph, CrewAI, or two hundred lines of your own async code. What ADK contributes is a tidy place to put them. Keep them in plain functions, as 8.25 recommends, and the framework becomes a decision you can revisit rather than one you are married to.

10.27 Interview drills

Framework questions are a trap in both directions. Reciting an API is not evidence of anything, and dismissing frameworks entirely usually means never having operated one. What an interviewer is listening for is whether you know which guarantees are structural and which are wishful.

1. What is the difference between adding an agent as a sub-agent and wrapping it as a tool?

A sub-agent is a transfer and a tool is a call. That sounds like a distinction about code structure, but the consequence is about conversation ownership: after a transfer, the sub-agent stays active and answers the next user turn too. With AgentTool, the specialist returns a result and the parent remains in control.

I check this by running two turns rather than one. A single-turn test cannot tell the two apart, which is exactly why the bug reaches production — every test asks one question, and real conversations do not.

My default is AgentTool, because it composes: I can consult two specialists and combine them, or do anything at all afterwards. I use sub-agents when stickiness is genuinely what I want — the conversation has changed subject and should stay there. There is also a cost angle: each transfer swaps the system prompt and toolset, which breaks prompt caching and re-sends the whole prefix uncached.

2. Your validator sets escalate on the callback context, and the loop still runs to max_iterations. Why?

Because the flag only reaches the loop if the callback also returns content or writes to state. ADK constructs the event carrying those actions in one of those two cases; with neither, the escalation is set on an object that is then thrown away. The loop never hears about it.

What makes this nasty is that nothing looks broken. The extra iterations regenerate an already-acceptable draft, so the output is fine and the only symptom is spend. I measured it at five iterations instead of one on a five-bound loop — five times the model calls for an identical result.

Two fixes. Either write something to state alongside the flag, which is what I do when the stopping condition is deterministic; or use the exit_loop tool, which sets the same flag from tool context where it cannot be discarded. And I would test it by reintroducing the bug deliberately to confirm the test actually fails — I have been caught by a test that passed either way because an unrelated state write was protecting it.

3. How would you stop an agent from fabricating figures in a generated report?

Not with a prompt. “Only use the provided sources” is a request to the component that fabricates, and nothing in the system notices when it is ignored.

The precondition is owning retrieval. If I use a provider's built-in grounded search, my application never holds the source documents, so I have nothing to compare a claim against and the check is impossible to write. So: run the search myself, keep every document in a ledger keyed by an identifier, and require each claim to cite identifiers.

Then validate in three tiers, ordered by cost. Citation integrity — does every cited id resolve? Literal grounding — does every number in the claim appear in the documents that claim actually cites? Both are regex-cheap and run on everything. Only what survives goes to an entailment check with a model.

Numbers get their own tier because they are the highest-signal and highest-damage case. Paraphrase makes prose hard to check mechanically, but a revenue figure either appears in the source or it does not — and it is the part a reader trusts most and verifies least.

4. Why bother with an entailment check if the cheap checks already passed?

Because a claim can be perfectly cited, contain only real numbers, and still say the opposite of its source. “Revenue grew to $1.2 billion” drawn from a document reporting that revenue fell to $1.2 billion passes both cheap tiers — the citation resolves and the number is genuine. Only reading it catches the reversal.

That is also the argument for ordering rather than choosing. Entailment is the only tier that costs a model call, so it runs last and only on survivors. A claim with a fabricated number is already known to be broken; sending it to a judge is spending money to re-learn something a regex established for free.

5. Your loop hits its iteration cap with problems outstanding. What do you publish?

Not the draft with its unsupported claims intact, which is unfortunately the default if nobody decides otherwise. I drop the specific claims that failed and publish the rest, marked incomplete. A section reading “revenue: not found” is more useful than one containing an invented number, because a reader can act on a gap and cannot act on a plausible falsehood.

I would also make the quarantine monotonic. Once a claim has failed, it stays out — otherwise a later pass can reintroduce it by rephrasing it in a way the validator happens to miss, and the final iteration undoes the work of all the earlier ones.

Before that, I would add stall detection. If the problem set is identical between two passes, the model is not converging and the remaining iterations are pure waste. Stop as soon as it stops shrinking.

6. Two parallel agents, and the second needs the first's output. How do you wire it?

I do not — the premise is the bug. Parallel branches are isolated, so a sibling's output_key is not visible during the block. The interpolation resolves to an empty string, no error is raised, and the model writes something plausible around the hole. It looks like it worked.

If B needs A's output then they are not independent, and putting them in a parallel block is a statement that they are. The correct shape is sequential, or fan-out then join: parallel legs gather independently, and a following stage reads all their outputs, which by then are all in state. If I genuinely need partial overlap, I split it — A alone, then a parallel block containing B and anything else that only needs A.

7. Are “skills” an Anthropic thing, or does Google ADK use them too?

Both, and that is the point. Anthropic originated and published the format, and ADK ships a full implementation — a skills module with the data model, a registry, local and cloud loaders, and a toolset that exposes them to a model. A SKILL.md written to the Claude conventions loads into an ADK agent unmodified.

The detail that settles it is that ADK's own source cites the agentskills.io specification — a vendor-neutral standard, not a competitor's docs. A framework does not reference an external spec for a format it considers someone else's property.

So it belongs in the same family as MCP and A2A: a protocol that started at one vendor and became shared. MCP standardises reaching a tool, A2A reaching another agent, and skills acquiring instructions. In all three cases the value comes from more than one implementation existing.

8. When is something a skill rather than a tool or another agent?

I ask what I am actually adding. A capability the model cannot perform itself — querying a database, sending mail — is a tool, because code has to execute. Knowledge or procedure — a refund policy, a house style, a checklist — is a skill. A separate reasoning context with its own model, tools or authority is an agent.

The tiebreaker I use: imagine pasting the content permanently into the system prompt. If that works but is wasteful, it is a skill and the win is context economy. If it is impossible because code must run, it is a tool. If it would conflict with the instructions already there, it is an agent, because what you need is isolation.

Skills matter because of progressive disclosure. Only names and descriptions go into the prompt up front; bodies load on demand. That is what makes fifty skills affordable, and it is really retrieval applied to instructions rather than documents.

9. When would you build a fully autonomous agent instead of a fixed pipeline?

Rarely, and only when the sequence genuinely cannot be specified in advance — when the task branches on what earlier steps discover. Autonomy is not the advanced option; it is what you fall back to when specification is impossible, and it costs every guarantee that specification provided.

What I give up: predictable cost, bounded latency, and debuggability. A failed run in a fixed pipeline means finding the stage that broke. A failed autonomous run means reconstructing a decision sequence that will never occur again.

Most tasks described as needing autonomy are unspecified pipelines. “Handle customer emails” sounds open-ended and is usually classify, retrieve, draft, check — four known stages, cheaper and far more debuggable as a sequential agent.

10. What bounds an autonomous agent when you cannot predict what it will do?

Four things, in order of leverage. A step cap, set from the 95th percentile of successful runs on my eval set plus headroom, rather than a round number that feels generous. A spend cap, because steps are a poor proxy for cost — one step that reads a long document outweighs twenty that do not.

Then the one that matters most: read-only by default. An autonomous agent with only read tools has a worst case of wasting money and being wrong. Add one side-effecting tool and the worst case becomes unbounded external damage. Every write is a separate, approved, idempotent tool.

And stall detection, because an agent that has stopped progressing still looks busy — it will re-read the same page and rephrase the same query until the budget is gone. One structural detail: I attach specialists as tools, never as sub-agents. A transfer would hand control away permanently and the loop's bounds would stop applying, which defeats the entire arrangement.

11. Run a multi-agent flow over 100,000 rows. What breaks first?

Three things, and I would address all three before starting. First, session reuse: each row needs its own session, or row 2 inherits row 1's history and starts producing summaries of supplier B that mention supplier A. It is a correctness failure with no crash, so at batch scale nobody notices.

Second, concurrency. A bare gather over 100,000 rows tries to call the model 100,000 times at once, hits rate limits in seconds, and the retry storm compounds it. A semaphore bounds in-flight work regardless of batch size.

Third, failure isolation. One row raising must not end the run — return_exceptions=True plus a per-row try turns a fatal error into a recorded one.

I would also do the arithmetic first. That pipeline is up to eight model calls per row, so 800,000 calls; at two seconds each with concurrency of eight that is over fifty hours. That calculation takes a minute and usually changes the design.

12. The batch job dies at hour three. What did you build so that this is survivable?

Results appended per row as they land, rather than collected in memory and written at the end. That single choice is the difference between a rerun that skips 30,000 completed rows and one that repeats three hours of work.

A dead-letter file holding the row, not just the error. Retrying then means filtering the failures, fixing the cause, and resubmitting exactly those rows. A dead-letter file of stack traces tells me what broke and leaves me to reconstruct what to re-run.

A lock around the writes, since concurrent coroutines otherwise interleave and produce lines that are half one JSON object and half another — a file that parses fine until it does not.

And before any of it, a 100-row sample that I read myself. Not the pass rate, the actual text. Systematic problems no validator catches — every summary opening with the same clause, one column consistently misread — are obvious in ten samples and invisible in an aggregate score.

Where this leaves you

You can now build a multi-agent system in ADK and say precisely which of its properties are guaranteed. The loop cannot exceed its bound. The specialist cannot silently keep the conversation, because you chose call-and-return deliberately. A claim without a resolvable citation does not reach the reader. One bad row in eighty thousand is a line in a dead-letter file rather than a failed overnight run.

None of those guarantees came from the model cooperating, and that is the thread running through the chapter. The framework contributes structure — a well-defined place to attach a validator, a cap that holds when the model is most confused, isolation between parallel branches. Everything that determines whether the output can be trusted is ordinary Python you wrote and could carry to another framework tomorrow. The evidence ledger, the tiered validator, the repair loop, the dead-letter queue: all of it is portable, and keeping it that way is what stops a framework choice from becoming a permanent one.

The failures worth remembering are the quiet ones. An escalation that is set and discarded, a sibling branch that reads an empty string, a session reused across rows — none of them raise, none of them log, and all of them produce output that reads correctly. They are found by an invoice, an auditor, or a customer. The defence is not more careful reading; it is measuring the behaviour you are relying on, and then breaking it on purpose to confirm your test noticed. That habit caught a wrong claim in the drafting of this very chapter, which is as good an argument for it as any.

That closes the course. Chapter 1 began with a model that could only produce text, and the arc since then has been about everything required to make that useful and safe: giving it knowledge it does not have, serving it at a cost that works, giving it tools that change the world, and — here — giving it structure that holds when it is wrong. The last part is the one that turns a demonstration into a system.