The first LLM feature ships as a direct call from one service to Bedrock. The tenth ships the same way, from ten services, with ten sets of credentials, ten retry policies, and no single place to see what anyone is spending. That is the moment a platform team discovers it needs an LLM gateway, usually a quarter after it would have been cheap to build one. A gateway is the shared front door every model call passes through, and it exists to own the four things individual teams keep reinventing badly: authentication, quota, routing, and audit.

The useful reframing is that a gateway is not an AI feature. It is the same control plane you already put in front of any shared, metered, expensive dependency. You would not let every service carry its own database admin password and pick its own connection limit. Model access is no different, except the bill grows faster and the failure modes are noisier.

What the gateway actually centralizes

The value is not in proxying the request. It is in owning the four cross-cutting concerns that do not belong in application code:

  • Authentication. Callers present a service identity to the gateway. The gateway holds the one credential that talks to Bedrock. No application ever sees a long-lived model key, and rotating that key is one change, not twenty.
  • Quota. Per-team and per-model token budgets live in one place. A runaway loop in one service hits its own ceiling instead of eating the shared tokens-per-minute limit and throttling everyone else.
  • Routing. The gateway maps a logical model name to a concrete deployment: which region, which provider, which fallback when the primary throttles. Applications ask for "the summarizer," not for a specific model ID in a specific region.
  • Audit. Every request and response passes one chokepoint, so cost attribution, prompt logging, and abuse detection all have a single seam to hook. Without it, cost-per-feature is an archaeology project.

Notice that none of these are model quality problems. They are platform problems that only look new because the dependency is a model.

Build vs LiteLLM vs API Gateway

There are three honest patterns, and the right one depends on how much of the above you actually need on day one.

Adopt LiteLLM (or a similar proxy)

An open-source proxy like LiteLLM gives you a unified API across providers, per-key budgets, and basic routing out of the box. It is the fastest way to get a real gateway that already knows about the four concerns. The trade-off is that you now run and patch a stateful service that sits on the critical path of every AI feature, and its opinions about routing and logging become your opinions. For most teams standing up their second or third LLM feature, this is the correct default. You buy the control plane instead of building it.

Put it on API Gateway plus Lambda

If you are already deep in AWS, Amazon API Gateway in front of a small Lambda authorizer and a proxy function gives you auth, usage plans, throttling, and access logs from primitives you already operate. Usage plans handle per-key rate limits; the authorizer handles identity; CloudWatch handles the audit trail. The cost is that model-specific logic, streaming, token counting, and provider fallback, is code you write and own. This pattern wins when your governance requirements are AWS-shaped and you would rather extend an existing platform than introduce a new dependency.

Build a bespoke service

Building your own is right only when your routing or audit requirements are genuinely unusual: complex model-selection policies, per-request data residency decisions, or logging that has to satisfy a regulator in a specific shape. Most teams that build from scratch are really rebuilding LiteLLM slower. Reach for this last, and only when you can name the requirement the off-the-shelf option cannot meet.

The failure mode a gateway prevents

Without a gateway, the failure is not dramatic, it is slow. Credentials spread. Each team sets its own timeout and retry, so a provider blip becomes a retry storm that pushes you through your token quota. Nobody can answer "what did the support-summarizer feature cost last month" without grepping logs across services. Then a security review asks who can call which model with what data, and the honest answer is "we are not sure," which is the answer that stops a launch.

A gateway turns all of those into configuration. That is the whole pitch: move the cross-cutting decisions out of twenty codebases and into one you can reason about.

Draw the line in the right place

Keep the gateway thin. It should handle identity, quota, routing, and logging, and stay out of prompt construction and business logic. The moment the gateway starts owning prompts, it becomes a shared library with a network hop, and every team is blocked on its release cycle. The durable design is a boring proxy with strong opinions about governance and no opinions about what you ask the model.

The takeaway

Every platform team that ships more than one LLM feature builds a gateway eventually. The only choice is whether you build it deliberately, early, while it is a small proxy, or accidentally, late, as a cleanup project after the credentials have spread and the bill has become a mystery. Adopt a proxy if you can, extend API Gateway if you are AWS-native, and build from scratch only against a requirement you can name. The gateway is not where AI happens. It is where the governance you were going to need anyway finally has a home.

Read this next

For the platform and infrastructure side of running a shared control plane like this, the cloud field notes live at ercan.cloud, and the hub is at ercanermis.com.