The gateway's guarantees are not properties of the gateway. They are properties of the transport the caller picked, and they get weaker with every step toward real-time. A plain request-response call is metered from the usage block the model actually returned. The same call with stream: true has its prompt tokens estimated whatever the policy says, and its completion tokens estimated too. A WebSocket connection cannot be load balanced across backends at all, because once established it is pinned one-to-one between client and backend. Part 2 built the infrastructure. This part puts an API on it, and the honest version of that API is one that tells each caller which guarantees it just gave up.

What a consumer team should have to know

Exactly three things: a base URL, a client credential, and a model name that means something to them rather than to Azure. Nothing about deployments, regions, providers, or which of those changed last Tuesday. That is the whole contract, and everything in this part exists to keep it true while the things behind it move.

API Management's unified model API is built for exactly this. Clients speak the OpenAI Chat Completions format, and the gateway translates to whichever backend format the target model actually uses. It supports two backend API formats, OpenAI Chat Completions and the Anthropic Messages API, and it does four things worth naming separately: standardizes the client-facing format independently of backends, applies one set of governance policies across providers, configures failover across providers, and decouples client-facing model names from backend model names using aliases.

The alias is the part that matters most here, and it is the direct continuation of Part 2. A model deployment pinned to gpt-4.1 version 2025-04-14 with NoAutoUpgrade is only useful if no application ever names it. Applications call chat-default. The alias maps chat-default to a deployment, and the day that mapping changes is a change to gateway configuration reviewed in a pull request, not a coordinated release across five teams.

Two caveats belong here rather than in a support ticket. The unified model API is in preview and rolling out; in the classic tiers early access runs through the AI Gateway Early release channel, which is a per-instance update setting. And its two supported backend formats do not cover everything the company runs. Mistral and Llama deployments in Microsoft Foundry are reached through their OpenAI-compatible chat completions surface, which the unified API can front as an OpenAI-format backend, but that is a different integration path from the native Anthropic translation, and it is worth writing down which models arrive by which route before anyone debugs a shape mismatch at 2am.

Routing: priority groups first, weights second

Behind an alias sits a backend pool, not a backend. API Management pools support round-robin, weighted, priority-based, and session-aware distribution, and the useful pattern for model traffic combines the first two: a priority group of Provisioned Throughput deployments that should absorb everything they can, and a lower-priority group of pay-as-you-go deployments that only sees traffic when the first group is unavailable. Within a group, weights split the load.

Pools are configured through the portal, the REST API, or an ARM or Bicep template. Part 2 chose Terraform, and this is precisely the case that part described as legitimate for the azapi escape hatch: one resource, a raw ARM body, sitting next to azurerm resources and referencing them by ID.

resource "azapi_resource" "chat_pool" {
  type      = "Microsoft.ApiManagement/service/backends@2023-09-01-preview"
  name      = "chat-pool"
  parent_id = azurerm_api_management.gw.id

  # azurerm has no backend-pool schema yet. Revisit when it does.
  body = {
    properties = {
      description = "PTU first, PAYG overflow"
      type        = "Pool"
      pool = {
        services = [
          { id = azapi_resource.be_ptu_west.id,  priority = "1", weight = "1" },
          { id = azapi_resource.be_payg_west.id, priority = "2", weight = "3" },
          { id = azapi_resource.be_payg_north.id, priority = "2", weight = "1" },
        ]
      }
    }
  }
}

Priority 1 is the PTU deployment: it is already paid for by the hour, so every token it can serve is a token that costs nothing extra. Priority 2 only receives traffic when priority 1 cannot take it, and the 3-to-1 weight between two regions reflects quota rather than preference. This is a cost topology expressed as routing, which is the kind of thing that is invisible in an architecture diagram and obvious on a bill.

The circuit breaker, and how much to trust it

A pool member that is failing should stop receiving traffic without a human deciding so. API Management exposes a circuit breaker as a property of the backend: rules define a failure count or percentage within an interval and the status code ranges that count as failure, and when the breaker trips the gateway stops calling that backend for the trip duration and returns 503 Service Unavailable to the client. If the backend sent a Retry-After header, the breaker can accept that value and wait exactly that long, which for a rate-limited model endpoint is far better than a fixed guess.

circuitBreaker = {
  rules = [{
    name = "ptu-5xx"
    failureCondition = {
      count    = 3
      interval = "PT1H"
      statusCodeRanges = [{ min = 500, max = 599 }]
    }
    tripDuration     = "PT1H"
    acceptRetryAfter = true
  }]
}

Two properties of this feature decide how you are allowed to reason about it. First, tripping rules are approximate: the gateway is distributed, instances do not synchronize breaker state, and each applies the rule on the information it has. A breaker configured to trip after three failures may in practice let considerably more through across a scaled-out instance. It is a protection mechanism, not a counter, and no SLO calculation should be built on its exact threshold. Second, it is not available in the Consumption tier, which is one more reason the tier decision in Part 2 was not about cost.

The 503 is worth designing for deliberately. A caller that receives 503 from the gateway cannot tell whether the model is down, the breaker is open, or the whole instance is unhealthy, and the three deserve different client behaviour. Adding a response header in the outbound policy that names which backend was tried, and logging the breaker state, turns an opaque 503 into something a consumer team can act on without opening a ticket.

Streaming, and the accounting it costs you

Every consumer team asks for streaming, and they are right to. Time to first token is the number a human perceives; total latency is the number a dashboard shows. Server-sent events over the same HTTP endpoint is the standard answer, the client sets stream: true, and nothing about the gateway's routing changes.

What does change is the meter. The token limit policy normally works from actual usage: with estimate-prompt-tokens="false" it reads the usage section of the model's response, which means a request can exceed the limit and be detected afterward, blocking subsequent requests until the window resets. With estimation enabled it counts prompt tokens from the API definition's schema before the call, trading a little accuracy for not burning backend quota to discover you were over.

Streaming removes the choice. When stream: true is set, prompt tokens are always estimated regardless of the policy setting, and completion tokens are estimated too. There is no configuration that makes a streamed request metered from ground truth at the gateway. For models that accept images the drift compounds: with streaming enabled, or with estimation on, the policy overcounts image tokens rather than taking the backend's count.

This is not a reason to refuse streaming. It is a reason to be precise about what the numbers mean, because Part 5 turns them into money:

  • Non-streamed HTTP: metered from the model's own usage. Exact.
  • Streamed SSE: prompt and completion tokens estimated at the gateway. Good enough for rate limiting, approximate for billing.
  • WebSocket: see below. Effectively out of band.

The control plane's answer is reconciliation rather than better estimation. Gateway metrics stay the real-time signal that enforces quotas, and the monthly attribution job Part 5 builds corrects against whatever authoritative usage the provider reports, with the delta tracked as its own metric. A drift that grows is a bug; a drift that stays flat at a few percent is the cost of streaming, and it should be written into the chargeback document rather than discovered by a product owner.

WebSockets, where the abstraction ends

Real-time voice and speech-to-speech traffic does not fit request-response, and API Management supports WebSocket APIs. The limitations are sharp enough that they change the architecture rather than decorating it.

WebSocket connections cannot be distributed or load balanced across multiple backends. Once established, each connection is maintained one-to-one between client and backend. Every routing decision in this post, the priority groups, the PTU-first topology, the weighted regional split, applies to the initial handshake and to nothing after it. A backend that degrades mid-session cannot be failed over by the gateway; the session has to end and the client has to reconnect. Any client on this path needs reconnect logic with backoff, and any capacity model needs to treat a WebSocket session as a pinned reservation rather than a stream of independently routable requests.

Three smaller constraints follow. A long list of policies cannot be applied to the onHandshake operation, including caching, CORS, body manipulation, and every validate-* policy, and policies inherited from a global or product scope are skipped at runtime rather than failing loudly. The set-header policy cannot change certain well-known headers, including Host, on handshake requests. And TLS validation is stricter than for HTTP APIs: for a WebSocket backend the gateway validates that the server certificate is trusted and that its subject name matches the hostname, where an HTTP API only requires trust. A backend with a mismatched certificate subject that has worked for years over HTTP will fail the moment it is fronted as a WebSocket API.

The practical consequence: the real-time path gets its own API, its own product, and its own quota, and the consumer documentation says plainly that it is not covered by the failover guarantees the chat endpoint has. Pretending otherwise is how an incident review discovers the difference.

Session affinity, and why to avoid needing it

Pools support session awareness through a cookie, with the client storing a Set-Cookie value and returning it so that subsequent requests reach the same backend. It exists for stateful server-side APIs, the Assistants API being the canonical example, where the client has to keep a session ID, extract a thread ID from the response body, and send the right cookie on the right call.

It works, and it is a constraint worth avoiding at design time. A session-affine call is a call that cannot be rerouted, which puts it in the same category as the WebSocket path: no failover, and a capacity model where a backend's load depends on which sessions landed on it rather than on how many requests arrived. Where a stateful API is genuinely required, keep it on a separate alias so its weaker guarantees are not silently inherited by the stateless traffic that makes up the rest of the volume.

Failure modes to watch

  • The 503 that means four different things. Breaker open, all pool members exhausted, gateway unhealthy, or a genuinely failing backend. Without a header or log line naming which, every one of them becomes a ticket.
  • Streaming drift treated as a bug. It is documented behaviour. The failure is not the drift, it is publishing an estimated number in a place that says invoice.
  • A preview feature in the critical path. The unified model API is in preview. That is acceptable for the translation layer only if the fallback, a direct per-provider API behind the same product and policies, already exists and is tested, rather than being designed during the incident.
  • Silently skipped policies on WebSocket APIs. Inherited policies that are unsupported on onHandshake are skipped at runtime. A security control assumed to be global is not global here, and nothing will tell you.

What Part 4 inherits

At the end of this part a consumer team has one endpoint, one format, and stable model aliases, with routing and failover behind them and a documented statement of which guarantees apply to which transport. What none of it handles is work that does not need an answer now: the nightly summarization of support tickets that would otherwise sit on the same synchronous path as a customer waiting for a first token. That is a queue, and it is next.

Read this next

For the infrastructure and platform side of running this at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.

References