The moment an application starts calling tools, the model's output stops being text to read and becomes an instruction to act, and every guarantee this series has built so far was about tokens. A framework can make that loop easy to write and, in the same motion, hide the throttling, reroute around the gateway's routing, and turn one metered request into eleven nobody planned for. Part 7 gave applications retrieval. This part is about what sits on top of it, and specifically about which jobs the framework should not be doing because the platform already does them.

Point the framework at the gateway, and stop there

The mechanics are trivial, which is why the discipline has to be explicit. An orchestration framework speaks the OpenAI Chat Completions format, the gateway exposes exactly that, so the integration is a base URL and a credential:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://apim-genai-prod.azure-api.net/llm/v1",
    api_key=gateway_token,        # Entra ID token, not a model key
    model="chat-default",         # gateway alias, never a deployment name
    max_retries=0,                # the gateway owns retry and failover
    timeout=60,
)

max_retries=0 is the line that matters, and it is the one people delete first. Frameworks ship with client-side retry, exponential backoff, and often provider fallback, all of which are reasonable defaults for an application talking directly to a model provider and all of which are actively harmful in front of a gateway that already does routing, load balancing, and circuit breaking.

Consider what client-side retry does to Part 3's circuit breaker. The breaker trips and returns 503 precisely so callers stop hammering a failing backend. A framework configured to retry 503 five times with backoff converts that signal into five times the load, arriving exactly when the backend is least able to take it. The same applies to 429: a token limit exists to shape demand, and a client that retries through it is not shaped. Retries belong in one place, and that place already exists.

The honest counterpart: a single retry on a connection-level failure, in the client, is fine. The rule is about retrying the gateway's own answers, not about network flakiness.

What the framework should still do

Having removed provider abstraction, routing, retry, and fallback from its job description, a framework is left with the parts that are genuinely tedious to hand-roll and carry no platform concern: prompt templating with typed inputs, tool schema generation from function signatures, output parsing into structured types, and the loop bookkeeping that turns a tool call into a follow-up message. That is a real amount of work and a reasonable reason to take the dependency.

What it should not become is the place where model choice lives. The alias in the code above is a gateway alias, resolved by the model catalog from Part 2. A framework's own model registry, its own fallback chains, its own provider list, all of that is a second catalog that will disagree with the first one on the day a model is retired.

Tools are the second surface, and they can share the first door

An agent's tools deserve the same treatment as its models: one governed entry point, per-tenant access, an audit trail. API Management can expose a REST API it already manages as a remote MCP server, publishing selected operations as tools that MCP clients call, available across the Developer, Basic, Standard, and Premium tiers including their v2 variants. Associating that MCP server with a product means tool access is managed through the same products and subscriptions as model access, which is the whole point: one identity, one quota story, one place to revoke.

Two limitations decide how far this goes today. API Management supports MCP server tools, but not MCP resources or prompts, for servers exposed from managed REST APIs. And MCP server capabilities are not supported in workspaces, which matters for exactly the isolated teams that Part 5 pointed at workspaces. A team that needs both a workspace gateway and MCP tool publishing has to pick one, and it is better to learn that while drawing the architecture.

Content safety on the prompt path

An agent loop widens the input surface: retrieved documents, tool outputs, and user text all end up in a prompt. The llm-content-safety policy routes content to Azure AI Content Safety before the model sees it.

<llm-content-safety backend-id="content-safety-backend" shield-prompt="true">
  <categories output-type="EightSeverityLevels">
    <category name="Hate" threshold="4" />
    <category name="Violence" threshold="4" />
  </categories>
</llm-content-safety>

Three attributes need reading carefully rather than copying. The threshold runs the way a tolerance runs, not the way a limit does: with threshold="4" the filter allows severity 0 through 3 and blocks 4 through 7, so raising the number raises tolerance and blocks less. It is the same inversion as the cache score threshold in Part 6, and it catches people the same way.

shield-prompt="true" turns on the check for adversarial user attacks, and defaults to false. For an agent that concatenates retrieved documents into its context, that default is the wrong one: prompt injection arriving through a document is the realistic threat, not a user typing an attack into a chat box.

enforce-on-completions, also false by default, extends the check to the model's responses when the policy sits in inbound. And window-size, which defaults to the 10,000 character Content Safety limit, is configurable only for responses; for requests the default window is always used. A very long retrieved context is therefore evaluated by the same windowing you do not control, which is an argument for keeping retrieval results bounded rather than for trusting the filter to scale with them.

What a loop does to every number in Part 5

A single user question that triggers four tool calls is not one request, it is five model invocations, each resending the growing transcript. Three consequences for the platform:

  • Per-minute quota stops being about user volume. It becomes user volume multiplied by average loop depth, and loop depth is a property of the application's prompt, which changes without a platform review.
  • A runaway loop is indistinguishable from load. The gateway sees well-formed requests inside quota. The application has to bound its own iterations, and the platform should publish that as a requirement rather than assume it.
  • Chargeback needs a correlation ID per user interaction, not per request, or the per-team numbers are true and useless. That ID belongs in the log record, which is where Part 5 put high-cardinality data for exactly this reason.

A per-request token ceiling at the gateway is a useful backstop and not a substitute. It caps the damage of one enormous call; it does nothing about a loop that makes two hundred small ones.

Failure modes to watch

  • Framework retry defeating the circuit breaker. The default is on. Turn it off deliberately and document why, or the platform's protection mechanism becomes a load amplifier.
  • A second model catalog inside the framework. It will disagree with the gateway's on retirement day, and the application will be pinned to a deployment name nobody knew it had.
  • Content safety threshold read as a limit. Higher is more permissive. A team tuning away false positives can disable the filter in practice while believing they tightened it.
  • shield-prompt left at its default. Off. On an agent that ingests retrieved documents, that is the injection path left open.
  • Unbounded loops. No gateway policy expresses "stop after six iterations". Only the application can, so it is a contract, not a control.
  • MCP plus workspaces. Unsupported together today. Discovering it after promising a team both is an architecture rollback.

What Part 9 inherits

Applications with orchestration, tools published through the same gateway, and content safety in the request path. All of it configured in policy XML, alias mappings, and product definitions that currently reach production the way any other change does, which is the part that has not been examined yet. Next: Azure DevOps, and what it means to gate a model version change behind an evaluation rather than a review comment.

Read this next

For the infrastructure and platform side of running this at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.

References