Llm-Gateway
Filed under
One Door to the Models, Part 10: Observability, Governance, and the Numbers
Logging prompts makes the gateway a processor of whatever users typed. Part 10 closes the series on traces, production evals, governance, and the scorecard.
One Door to the Models, Part 9: Eval Gates and Model Retirement
An approval that times out marks the stage skipped, not failed. Part 9 makes the eval a real gate and turns a model retirement into a query, not a project.
One Door to the Models, Part 8: Orchestration on Top of the Gateway
An orchestration framework that retries and reroutes is competing with the gateway. Part 8 keeps the framework thin and puts tools under the same identity.
One Door to the Models, Part 7: Azure AI Search or a Vector Database
Azure AI Search bills capacity by SKU, not by vectors, and the vector index is capped by tier memory. Part 7 decides retrieval on limits, not features.
One Door to the Models, Part 6: Semantic Caching and Its Failure Modes
A semantic cache is a correctness surface, not just a cost lever. Part 6 tunes the threshold, isolates tenants, and handles the day the cache is gone.
One Door to the Models, Part 5: Identity, Quota, and Chargeback
You get five custom metric dimensions, and time series multiply. Part 5 builds tenant identity, quota, and a chargeback model that survives cardinality.
One Door to the Models, Part 4: Async Work Off the Request Path
Batch inference is a second door: a file goes in, a job runs, a file comes out, and no gateway policy is in the path. Part 4 keeps the accounting honest.
One Door to the Models, Part 3: The Provider Abstraction and Streaming
Set stream to true and token counting becomes estimation. Open a WebSocket and load balancing stops existing. Part 3 maps guarantees to transports.
One Door to the Models, Part 1: The Case for a Central LLM Gateway
A company runs five GenAI apps and cannot say what any of them cost. Part 1: the scenario, build versus buy on Azure, and the gateway this series builds.