A large corporate IT organization has five generative AI applications in production, eleven model deployments spread over three Azure subscriptions, a monthly bill somewhere around 38,000 euros, and no reliable way to answer which of the five spent it. Nothing is broken. Every app works, every team shipped, every security review passed at the time. What does not exist is a single place where a token gets counted, a caller gets identified, or a model gets retired, and that absence is what this ten-part series builds: one governed GenAI gateway on Azure that every application goes through, and nothing goes around.

The company, and what it is actually paying

Call it what it is: an established corporate IT organization, roughly 400 engineers across about 40 product teams, most workloads on Azure, a platform group that already owns the landing zones and the CI templates. Generative AI arrived the way it usually does, from the edges inward. A customer-service assistant shipped first because a product owner had budget. Then an IT-operations summarizer, an internal code assistant, a retail knowledge search over product documentation, and a marketing copy tool. Five teams, five decisions, five direct integrations against Azure OpenAI.

Each of those decisions was correct in isolation. Together they produce a specific and now familiar shape:

  • Eleven model deployments across three subscriptions. Two of them are duplicates of the same model in the same region, created because a team could not get quota attached to the resource it already had.
  • Around 38,000 euros a month of model spend that cost management reports at subscription granularity, which means it can tell finance the total and cannot tell any product owner their own number.
  • Four of the five apps hold a key in application settings. One held it in a repository until a secret scanner found it. No app can be individually revoked without an engineer editing that app's configuration.
  • A 60 day model retirement notice landed in an email, and the platform group spent two days grepping repositories to work out which applications would break, because nothing recorded which app called which model version.

Then the incident that actually forced the decision. A batch job in the marketing copy tool consumed most of the tokens-per-minute quota on a shared deployment for about forty minutes. The customer-service assistant, sharing that deployment, started returning HTTP 429 to real customers. Neither team did anything wrong. There was no mechanism in the path where "one caller must not exhaust the quota another caller depends on" could be expressed, because there was no shared path at all.

That is the honest case for a gateway. Not "central control is good architecture". It is that quota, cost attribution, credential rotation, and model lifecycle are all properties of the call, and if no component sees every call, none of them can be enforced anywhere except by convention.

What the per-app pattern actually costs

Direct integration is genuinely cheaper on day one. One SDK client, one key, one endpoint, no platform team in the critical path. The cost arrives later and lands somewhere other than the team that chose it, which is exactly why it keeps getting chosen.

The compounding costs are three, and they are all the same cost wearing different clothes. Change cost: a model retirement, a price change, a region migration, or a provider outage has to be handled once per application, by five teams with five backlogs and five different levels of interest. Attribution cost: spend is visible per resource and invisible per consumer, so nobody can be given a budget, which means nobody can be given a limit. Governance cost: every new control, prompt logging, PII redaction, an EU AI Act transparency requirement, has to be implemented five times and verified five times, and the review effort scales with the number of applications rather than the number of controls.

A gateway does not remove those costs. It moves them to one place where a platform team can pay them once. That is the whole argument, and it is worth being suspicious of any richer version of it.

Build versus buy on Azure

The team evaluated three shapes, and the answer is not the one an engineer wants it to be.

Azure API Management as the AI gateway. API Management ships a set of capabilities explicitly aimed at language model traffic, and the list is longer than most people assume before they read it. The llm-token-limit policy enforces tokens-per-minute or a token quota per hour, day, week, month, or year against any counter key, including a subscription key, a caller IP, or an arbitrary policy expression, and it can precalculate prompt tokens on the gateway so an over-limit request never reaches the backend at all. The llm-semantic-cache-store and llm-semantic-cache-lookup policies do vector-similarity caching of completions against Azure Managed Redis or another RediSearch-compatible cache. The llm-emit-token-metric policy emits token counts to Application Insights with custom dimensions, which is the raw material for per-consumer attribution. Backends support round-robin, weighted, priority-based, and session-aware load balancing, and a circuit breaker whose trip duration honours the backend's own Retry-After header, so a pay-as-you-go endpoint can absorb overflow from a saturated Provisioned Throughput Unit deployment without anyone writing retry code.

Provider coverage matters here and is easy to get wrong. API Management mediates language model APIs conforming to the OpenAI Chat Completions and Responses schemas, the Anthropic Messages API (in the v2 tiers), and the Google Vertex AI API, against models deployed in Microsoft Foundry or non-Microsoft providers including Amazon Bedrock. A unified model API, in preview at the time of writing, exposes several backends behind a single OpenAI-compatible endpoint and translates formats, so one governance policy set applies across providers. Anthropic Claude, Mistral, and Llama are reachable through this layer rather than requiring a separate integration each.

An open-source proxy on AKS. LiteLLM and similar projects give you provider translation and routing in a container you fully control, with a faster path to a feature you want and no dependency on a preview. You inherit an on-call rotation for a component that now sits in front of every AI request in the company, plus the identity, network, and audit work that API Management already has certified.

A bespoke FastAPI service. Maximum control, and the only option where every failure is yours. Reasonable for a component with genuinely unusual requirements, which a token-counting reverse proxy is not.

The decision this series follows: API Management is the data plane, and the things API Management does not do get built as a small Python control plane on AKS. That is the split worth internalizing, because it is where most gateway projects go wrong. Teams either try to express everything as policy XML, which produces an unmaintainable artifact nobody can test, or they route the traffic through their own service and reimplement rate limiting badly. The gateway carries the request. The control plane carries the state: which models exist, who may call them, what a call costs, and what happens when a model is retired.

What you still build

Five things, and every one of them is a later part of this series.

  • A model catalog and lifecycle. Which logical model names exist, which physical deployments back them, which are deprecated and when they disappear. Applications must call chat-default, never a deployment name, or the 60 day retirement email becomes a 60 day migration project again.
  • Tenant identity and onboarding. A team gets a client credential, a quota, a model allowlist, and a cost centre, provisioned as code rather than as a ticket.
  • Cost attribution and chargeback. Token metrics with a tenant dimension are only the input. Turning them into a per-team monthly figure that finance accepts is a data pipeline with its own tests.
  • Evaluation gates in the pipeline. Promoting a new model version is a deployment. A deployment that changes model behaviour without an eval result attached is an outage waiting for a customer to find it.
  • Async and long-running work. Batch summarization of support tickets does not belong on the same synchronous path as a customer waiting for a first token. That separation is a queue, not a timeout setting.

Target architecture

The platform is deliberately boring and deliberately additive. If the control plane is down, the data plane keeps serving traffic on its last known configuration, because a gateway that fails closed on a control-plane outage is a company-wide outage.

graph TD
  subgraph consumers["Consumer apps"]
    A1["customer-service assistant"]
    A2["IT-ops summarizer"]
    A3["code assistant"]
    A4["retail knowledge search"]
    A5["marketing copy tool"]
  end

  subgraph gw["Data plane: API Management"]
    POL["policies:
llm-token-limit, semantic cache,
emit-token-metric, JWT validate"]
    LB["backend pool:
priority + weighted, circuit breaker"]
  end

  subgraph cp["Control plane: Python on AKS"]
    CAT["model catalog"]
    TEN["tenant registry"]
    COST["cost attribution job"]
  end

  subgraph models["Model layer"]
    FDRY["Microsoft Foundry:
GPT deployments, PTU + PAYG"]
    ANTH["Anthropic Claude"]
    OSS["Mistral, Llama"]
  end

  ID["Microsoft Entra ID"]
  RED["Azure Managed Redis
semantic cache"]
  SRCH["Azure AI Search
retrieval"]
  MON["Azure Monitor +
Application Insights"]

  A1 --> POL
  A2 --> POL
  A3 --> POL
  A4 --> POL
  A5 --> POL

  ID -. "OAuth2 client credentials" .-> POL
  POL --> LB
  POL <--> RED
  LB --> FDRY
  LB --> ANTH
  LB --> OSS
  POL -- token metrics, prompt logs --> MON
  CAT -. "backend + policy config" .-> gw
  TEN -. "subscriptions, quotas" .-> gw
  MON --> COST
  A4 -. "grounding data" .-> SRCH

Two properties of that diagram are load-bearing. First, every arrow into the model layer originates inside API Management, which means network rules on the Foundry resources deny everything else: the gateway is not a suggestion, it is the only route with a path. Second, the control plane writes configuration into the gateway and reads telemetry out of Azure Monitor, but never sits in the request path. It can be redeployed at noon.

The stack, in one place

What the ten parts touch, so the shape of the series is visible from here. Infrastructure as code in Terraform, with the Bicep and ARM comparison made honestly in Part 2 rather than assumed away. Docker images and AKS for the control plane. Azure DevOps for pipelines, versioning, and release management. Python everywhere the logic lives. Microsoft Entra ID with OAuth2 client credentials and API keys where a legacy caller cannot do better. Azure AI Search and a dedicated vector database for retrieval, compared rather than declared. Azure Managed Redis for semantic caching. Azure Monitor and Application Insights for telemetry, alongside LLM-specific observability in LangSmith and Openlayer, and Dataiku where a data science team already lives. LangChain for orchestration in the one part where a framework earns its place and the parts where it does not. Streaming over server-sent events, WebSockets where a session is genuinely bidirectional, and a message queue for everything that does not need an answer now.

The ten parts, in order: the case for a gateway; Terraform against Bicep and ARM; the provider abstraction and streaming; async and event-driven work; identity, quota, and chargeback; semantic caching; retrieval with Azure AI Search against a real vector database; orchestration on top of the gateway; CI/CD and model lifecycle in Azure DevOps; and observability with AI governance.

What good looks like

The final part comes back to these, which are the four numbers from the toil inventory made measurable:

  • Every product owner can see their own token spend for the current month, within a day of it being incurred, without asking the platform team.
  • No application holds a long-lived model credential. Revoking one team's access is a configuration change in the tenant registry, not an edit to that team's app settings.
  • A model retirement notice produces a query, not an investigation: the catalog already knows which tenants resolved to that deployment in the last 30 days.
  • One tenant cannot exhaust another tenant's quota, verified by replaying the 429 incident above as a load test against the gateway.
  • Median added latency from the gateway stays under 60 milliseconds at the 95th percentile for non-cached calls, and semantic cache hits return in a fraction of a model call.

Failure modes to watch from day one

Three, and they are worth naming before any Terraform is written. The gateway becomes a single point of failure the moment it is the only route, which is the point, so its availability target has to be higher than that of the most demanding application behind it, and its deployment has to be zone-redundant from the first apply rather than after the first incident. Policy sprawl arrives quietly: policy XML is code, it is executed on every request, and it is easy to write in a place where nobody reviews it. It belongs in version control with the rest of the platform, applied by pipeline, never edited in the portal. And the bypass path defeats everything else, because a single team that keeps a direct key to a Foundry endpoint makes every attribution number and every quota guarantee approximate. That is a network rule and an alert, not a policy document.

Read this next

For the infrastructure and platform side of running this at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.

References