AI Agents and Tool Calling

8.1 A question retrieval can't answer

"What is your returns policy" is a lookup. "Cancel order 88421 and refund me" is not — it requires the model to change something outside itself, and that single distinction is what this whole chapter is about.

Chapter 5's agentic RAG gave a model a search tool and let it decide when to call it again. Every one of those tools shared a property worth naming explicitly:

Nothing about how a model decides to call a tool changes based on this distinction. The model doesn't know or care whether search_products or cancel_order is "riskier" unless you tell it so. What changes is everything downstream of the call: how carefully you validate the arguments, whether a human needs to approve it first, how you handle a duplicate call, and how much damage a confused model can do before something notices.

8.2 How tool calling actually works

Worth being precise about this, because "the model calls the tool" is a convenient simplification that hides an important fact: the model never calls anything. It only ever writes structured text describing what it would like called.

1. You send: [conversation so far] + [list of tool schemas]

2. Model replies: "call cancel_order with {order_id: '88421', reason: 'customer request'}"
   (structured, not prose — this is the part your code parses)

3. Your code: actually calls cancel_order("88421", "customer request")
              validates the model's arguments FIRST (8.4), executes, catches errors (8.7)

4. You send back: the tool's result, tagged so the model knows which call it answers
   Model then either replies to the user, or requests another tool call — repeat.

The exact wire format differs slightly by provider — OpenAI's API calls this tools and returns a tool_calls array; Anthropic represents it as a tool_use content block and expects a matching tool_result block back. The shape underneath is the same exchange described above in every case, which is exactly the redundancy that motivates the standardisation in Part C of this chapter.

8.3 The agent loop, generalised

Section 5.23's loop — think, call a tool, observe the result, repeat — doesn't change shape when the tools have side effects. What changes is what has to be true before the loop is allowed to take a step.

while not done:
    response = model.generate(conversation, tools=available_tools)

    if response.wants_tool_call:
        call = response.tool_call
        if not passes_validation(call):           # 8.4 / 8.7
            conversation.append(error_result(call))
            continue
        if requires_approval(call):                # 8.5 / 8.21
            if not await_human_approval(call):
                conversation.append(denied_result(call))
                continue
        result = execute(call)                     # the only side-effecting line
        conversation.append(result)
    else:
        done = True
        return response.text

Every line added to chapter 5's version of this loop is a checkpoint that exists specifically because execute(call) might not be reversible. None of them are needed for a read-only tool — a search that runs when it shouldn't have is a wasted step, not an incident.

8.4 Writing schemas a model can use

A tool's description is not documentation for a human developer. It is the only information the model has about when and how to use the tool, and a vague one produces wrong calls no amount of downstream validation fully catches.

8.5 Read-only versus side-effecting tools

Section 8.1 named the distinction. This section makes it a property you actually declare per tool, because every safety decision later in the chapter reads from it.

The Model Context Protocol formalises exactly this, and its vocabulary is worth adopting whether or not you use MCP itself. Each tool can carry annotations describing how it behaves:

Why idempotency is the property that matters most

Of the four, idempotentHint is the one that most directly determines what your code has to do. Agents retry — on a timeout, on a network blip, on a malformed response. If a retried call is idempotent, retrying is free. If it isn't, a retry is a second refund.

8.6 Parallel calls and dependencies

A model can request several tool calls in one turn. Whether that's a latency win or a correctness bug depends entirely on whether the calls actually depend on each other — and the model is not reliably able to tell.

Most providers support emitting multiple tool calls in a single response. When the calls are genuinely independent, this is straightforwardly good: three lookups that would have taken three sequential round trips take one, and chapter 7's ray.wait() or an ordinary asyncio.gather executes them concurrently.

8.7 Handling tool failures

Tools fail. The interesting question is not how to prevent that, but what the model sees afterwards — because a tool error is one of the few situations where the model can genuinely recover on its own if you let it.

There are three distinct failure kinds, and conflating them is what produces agents that spin uselessly:

1. Invalid arguments — the model's mistake, and recoverable A malformed order ID, a missing required field, a string where a number belongs. Your validation catches this before execution. Return a specific error explaining what was wrong, and the model very often fixes it on the next turn.
2. Legitimate business rejection — not an error at all "This order is outside the 30-day return window." Nothing malfunctioned; the answer is no. This should read as a normal tool result, not an exception, or the model treats a correct answer as something to retry around.
3. Infrastructure failure — not the model's problem Timeout, connection refused, 500 from an upstream service. The model can do nothing useful with this. Retry it in your code with backoff, and only surface it to the model if retries are exhausted.

8.8 The N×M integration problem

Section 8.2 mentioned that every provider expresses the same tool-calling exchange in a slightly different wire format. That inconsistency is annoying at one integration and becomes a genuine architectural problem at twenty.

Suppose your organisation has built tool integrations for orders, inventory, refunds, the warehouse system, and the CRM. Now suppose you have three applications that want to use them, each built against a different model provider. Without a shared standard, every application re-implements every integration in that provider's format:

8.9 MCP: tools, resources, prompts

An MCP server exposes three kinds of thing. The distinction between them is not bureaucratic — it's about who decides when each one gets used.

from mcp.server import MCPServer

mcp = MCPServer("Orders")

@mcp.tool()
def get_order_status(order_id: str) -> dict:
    """Look up an order's current status, items, and shipping address by order ID.

    order_id is the 5-8 digit number from the receipt, not a product SKU.
    """
    return orders_db.fetch(order_id)

The docstring becomes the tool description the model sees, and the type hints become the inputSchema. Everything section 8.4 said about writing descriptions carefully applies directly to that docstring — it is not a comment, it's the prompt the model reads when deciding whether to call this.

8.10 Sampling and elicitation

Two newer MCP primitives that invert the usual direction: instead of the client asking the server for something, the server asks the client. Both exist to keep a person in the loop, which makes them directly relevant to everything in Part F.

8.11 Beyond ReAct

ReAct — reason, act, observe, repeat — is the default agent loop and section 5.23 covered it. It has a specific weakness that shows up as soon as tasks get longer, and two alternative shapes address it.

8.12 Short-term versus long-term memory

"Memory" gets used for two completely different things in agent systems. They have different mechanisms, different failure modes, and one of them is a solved problem while the other mostly isn't.

8.13 A scenario needing both

Planning and memory are easy to describe separately and easy to under-appreciate until a task needs both at once. Here is one that does.

Nothing about this is answerable from the message alone. It requires knowing what "last time" was, planning a multi-step resolution, and executing side-effecting tools in an order that matters.

1. Long-term memory supplies the missing referent "Same as last time" is meaningless without retrieving the prior interaction: a damaged-on-arrival replacement, resolved by refund plus a prepaid return label. Without persistent memory the agent must ask, which is precisely the experience the customer is signalling frustration about.
2. An explicit plan makes the sequence inspectable Verify the order and damage claim, issue a refund for the item, generate a prepaid return label, waive the postage charge, confirm to the customer. Two of those five steps spend money. Producing the plan first means the whole strategy can be checked against policy before any of it executes — the safety property from 8.11.
3. Ordering is load-bearing Generate the return label before confirming; confirm before the refund settles and the customer's expectation is set wrongly. Section 8.6's rule applies: these side-effecting steps run one at a time, each checked, not batched in parallel.
4. Working memory has to survive the whole run The order total retrieved in step one is what step two refunds. Five tool results later, that number is mid-context where attention is weakest — so the plan carries it forward explicitly rather than trusting the model to find it again.
5. The interaction is written back to long-term memory So that the next "same as last time" resolves correctly — with provenance and a timestamp, per 8.12's warning.

8.14 Context engineering

Section 8.12 gave four tactics for keeping working memory under control. This section is about treating those tactics as a single engineering discipline with a name, a budget, and a policy — because on a long-running agent, what occupies the context window determines behaviour more than the prompt does.

The eviction policy is the design

Once you accept the window is finite, the real question is not how to fit more in but what to drop first. That ordering is a design decision, and most agents make it by accident.

ALWAYS RESIDENT          ~4k   system prompt + tool schemas (cacheable, never edited)
                         ~1k   the original request, restated verbatim near the tail
PROTECTED TAIL          ~20k   last 3 tool results in full
COMPACTED MIDDLE        ~80k   older turns; compacted from source at 70% full
HEADROOM                ~23k   next tool result + the model's reply

TRIGGER   compact when total > 90k, not every turn (prefix cache)
NEVER     compact the system prompt, the goal, or the last 3 results

8.15 Harness engineering

Section 8.3 said the loop is the easy part and that the difficulty lives in what surrounds it. That surrounding machinery has a name, and treating it as a designed artefact rather than glue code is what separates an agent that demos from one that works.

The tool surface is a user interface

Section 8.4 covered writing schemas the model can use. The harness view is one level up: not "is this tool well described" but "is this the right set of tools, at the right granularity."

Observation formatting: what the model sees after acting

WEAK    {"error": "ValidationError", "code": 422}
        -> model retries the identical call, fails again, loops

STRONG  Error: 'date_from' must be ISO format (YYYY-MM-DD).
        You sent '90 days ago'. To search recent orders,
        compute the date first or omit date_from for all orders.
        -> model corrects on the next step

Verification inside the loop

8.16 Why agents fail at long horizons

An agent that handles a four-step task reliably will not necessarily handle a forty-step one at all. The failure is not linear, and the reasons are specific enough to design against — which is the point of naming them.

per-step      5 steps    10 steps    20 steps    50 steps
  99%          95.1%       90.4%       81.8%       60.5%
  95%          77.4%       59.9%       35.8%        7.7%
  90%          59.0%       34.9%       12.2%        0.5%

The four failure modes

8.17 When to split into agents

Section 5.23 already warned that multi-agent is usually premature, and gave the arithmetic: four agents at 90% reliability each is a system at 66%. That warning stands. This section is about the cases where splitting genuinely earns its cost.

The honest default remains one agent with a well-described toolbox. Most "we need multiple agents" instincts are really a single agent with confusing tool descriptions — which 8.4 fixes far more cheaply than an orchestration layer does. Three situations genuinely justify the split:

8.18 Agents as Ray actors

Chapter 7 built exactly the machinery a multi-agent system needs, for a different reason. An agent that holds a conversation, remembers state, runs concurrently, and must be reachable by name is — precisely — a named async actor.

@ray.remote
class RefundAgent:
    """Holds the refund credential. No other agent does."""

    def __init__(self, gateway_handle):
        self.gateway = gateway_handle        # the vLLM gateway from 7.7
        self.tools = [issue_refund, get_order]   # narrow, deliberate toolset

    async def handle(self, task: dict) -> dict:
        conversation = build_conversation(task)
        for _ in range(MAX_STEPS):           # the iteration cap from 5.23
            reply = await self.gateway.generate.remote(conversation, tools=self.tools)
            if not reply.wants_tool_call:
                return {"result": reply.text}
            conversation.append(await self.execute_checked(reply.tool_call))
        return {"result": "escalated", "reason": "step limit reached"}

refund_agent = RefundAgent.options(
    name="refund_agent", namespace="agents",
    get_if_exists=True, lifetime="detached", max_concurrency=50,
).remote(gateway)

8.19 A worked orchestration scenario

One request, traced through a supervisor and two specialists, showing where each chapter's machinery actually engages.

Two genuinely different jobs in one message: a cautious warranty-and-replacement transaction, and an open-ended product recommendation. This is 8.17's reason 1 and reason 2 arriving together.

1. The supervisor decomposes, and does not act A named async actor holding no side-effecting tools at all. It splits the message into a warranty task and a recommendation task. Holding no dangerous tools is deliberate — the component doing free-form interpretation of user text is the one most exposed to injection.
2. The recommendation task dispatches immediately, in parallel It's read-only, so per 8.6 there's no reason to serialise it. The supervisor submits it to an ActorPool of catalogue-search agents and doesn't wait — chapter 7's fan-out pattern.
3. The warranty task goes to the specialist, sequentially WarrantyAgent checks the purchase date and failure description, calls check_warranty (read-only), and determines the claim is valid.
4. The replacement crosses a threshold and triggers approval create_replacement_order is side-effecting, non-idempotent, and above the auto-approve limit. Per 8.5 and 8.21 it requires confirmation — surfaced through elicitation (8.10), carrying an idempotency key so a retry can't duplicate the order.
5. Results converge, and the supervisor composes one reply Recommendations came back while the warranty path was still waiting on approval. The supervisor merges both into a single coherent answer. Handoff payloads are explicit, per 8.17's warning.

8.20 Why side effects raise the stakes

Section 5.23's four guardrails bound how much an agent can spend. Once tools act on the world, you also need to bound how much it can do — and those are different problems with different controls.

8.21 Human-in-the-loop approval

The control that actually stops bad actions before they happen. The design question is never whether to have approval gates — it's where to put them, because a gate on everything is the same as no gate at all.

Deciding what needs a gate

8.22 Cost and spend controls

Two entirely different budgets share the word "cost" in agent systems: tokens burned, and money moved. Both need caps, and confusing them leaves one of them unbounded.

Where to put the limits

1. Per-run token budget A hard ceiling on total tokens for one task, checked between steps. When exceeded, stop and escalate rather than continuing — chapter 5's guardrail, still correct.
2. Per-session action budget Total real money one conversation may move. A support session that has already refunded £400 should require approval for the next pound regardless of thresholds.
3. Per-user, per-day limits Catches the case a single session cannot see: the same customer opening eight conversations to obtain eight refunds. Session-scoped budgets are blind to this by construction.
4. Global circuit breaker An aggregate rate across all sessions, with an automatic halt. If total refunds in an hour exceed several times the normal rate, stop the capability entirely and alert someone.

8.23 Observability and evaluation

Agents fail in ways single model calls don't: the final answer looks fine and step four was wrong, or every step was individually reasonable and the sequence made no sense. You cannot debug that from outputs alone.

Metrics that mean something for agents

8.24 The framework landscape

Everything so far has been mechanism, deliberately framework-free, because the mechanism is what you're accountable for. This section is about which library packages that mechanism for you, and the honest answer to "which one" is narrower than the marketing suggests.

The differences that are actually real

Having said no capability gap exists, four differences genuinely affect what you can build without fighting the library:

8.25 Google ADK in detail

ADK is worth a closer look not because it can do something the others can't, but because the shape of what it ships tells you what Google thinks the hard parts of production agents are — and that assessment is largely correct.

Agent types: composition rather than prompting

ADK's central idea is that orchestration should be structural. Rather than instructing one model to "first do A, then B," you compose agents whose control flow is code:

The operational surface

A2A: agents calling agents across boundaries

8.26 Evaluating multi-agent systems

Section 8.23 argued for grading the trajectory rather than the destination. Multi-agent systems make that harder in a specific way: when a five-agent run fails, the failing agent and the responsible agent are frequently not the same one.

Four levels, and what each can and cannot see

How ADK operationalises this

ADK's evaluation module is worth studying even if you never adopt it, because its metric taxonomy is a concrete answer to the levels above. Its named metrics include:

Simulated users, and the trap in them

8.27 Key takeaways

8.28 Interview drills

Agent questions separate people who have shipped one from people who have read about them. The tell is almost always whether the answer engages with what happens when a tool call is wrong.

Expand all

Where this leaves you

You can now design an agent that acts rather than only answers: tool schemas a model can actually choose between, validation and idempotency in the gap between request and execution, a protocol layer that stops every application re-implementing the same integrations, planning and memory sized to the task, specialists whose credentials are genuinely out of reach of the components most exposed to manipulation, and budget controls at every level from a single call to the whole system.

The thread running through the chapter is one question asked repeatedly: what happens when this step is wrong? Chapter 5's agentic RAG could mostly shrug at that question, because a bad search is a wasted search. Once a tool moves money or sends something to a real person, the answer has to be concrete at every step — and the design work is almost entirely in making sure it is.

From here, chapter 9 steps back from the model layer entirely and covers the API design that fronts everything built so far: the concrete choices for exposing a React frontend to endpoints that include long-running LLM inference, why FastAPI became the default for that job, and the HTTP-level concepts — streaming, auth, versioning, idempotency — that this chapter's tool layer and chapter 6's serving layer both sit behind.