The moment the gateway logs prompts and completions, it holds whatever a customer-service agent's user typed into a chat box, which may include an address, an account number, or a complaint about their health. Every other decision in this series traded cost against latency or control against convenience. This one trades debuggability against a data protection obligation, and it is the last one because it is the one that needs everything else already in place. Part 9 got the platform shipping. This part closes it: what to record, what to watch, what the regulation asks, and whether the numbers from Part 1 actually moved.

Three layers, and the one you build yourself

The platform now emits from three places, and conflating them is how teams end up with three dashboards that disagree.

  • Gateway telemetry. Token metrics into Application Insights with the tenant, model alias, and environment dimensions from Part 5, plus request logs into Azure Monitor. This is the authoritative view of who called what and what it cost, and it is the only layer that sees every synchronous request.
  • Application telemetry. The orchestration layer's own traces: which tools ran, how many loop iterations, which retrieved documents made it into the prompt. LLM-specific tools like LangSmith or Openlayer live here, and a data science team already using Dataiku will want its own view of the same data. None of these see traffic that does not go through the application that emits them, which is worth saying out loud whenever someone proposes one as the platform's source of truth.
  • Model and safety signals. Content safety blocks, refusals, filter categories triggered. These come from the gateway policy and belong with the gateway layer, not the application one, because the whole point is that they hold even when the application misbehaves.

The layer the platform team owns and nobody else will build is the join: one correlation ID that follows a user interaction across all three, so a single question that triggered four tool calls and one content safety block is one thing in a query rather than six unrelated records. Part 8 put that ID in the log record. This is what it was for.

What to log, and what not to

Logging prompts and completions is genuinely useful: it is how you debug a bad answer, build the golden set from real traffic, and prove what the system said when someone disputes it. It also turns the log store into a repository of user-generated content of unknown sensitivity, replicated wherever those logs go.

The workable position is per-product rather than global, and it is a decision the consuming team makes with the platform rather than one the platform makes for everyone:

  • Always log the metadata: tenant, alias, resolved deployment and version, token counts, latency, status, cache hit, content safety outcome, correlation ID. None of it is user content and all of it answers the operational questions.
  • Log content by exception, with a retention shorter than the metadata's, scoped to products that opted in, and with the subscription key stripped from the request as Part 5 required.
  • Sample rather than capture everything where content logging is on. A representative sample debugs as well as a full capture for most purposes and reduces the exposure proportionally.

Write the retention periods and the legal basis into the same document as the chargeback model. A platform that can say precisely what it keeps, for how long, and why, survives a data protection review. One that cannot will be told to turn logging off entirely, which costs far more.

Evals in production, not just in the pipeline

Part 9's eval gate answers "is this change safe to ship". It does not answer "is the system still good", because production traffic is not the golden set and drifts away from it continuously.

Two mechanisms, and they answer different questions. Scheduled evals run the golden set against production configuration on a timer, which catches drift in things nobody deployed: a model version that moved despite the pin, an index that reindexed differently, a retrieval path that got slower. Sampled online evaluation scores a small share of real traffic, usually with a model-as-judge, and is the only way to notice a class of failure the golden set never contained, because the golden set is by definition made of failures you already knew about.

Both need somewhere to put the result, and it is the same place as everything else: a score with the tenant and alias dimensions, on the same dashboard as spend and latency. A quality number that lives in a separate tool gets looked at during incidents and ignored the rest of the time.

Governance, and what the regulation actually asks

Under the EU AI Act, most of what this company runs is limited-risk rather than high-risk, and the obligations that follow are mostly about transparency and traceability rather than certification. That is not a reason to treat governance as a documentation exercise, because the questions an auditor asks map almost exactly onto things the platform either records or does not.

  • Which models are in use, in which applications, in which versions? The model catalog and alias map answer this. Without them it is a survey.
  • What data went to which provider, and where was it processed? The backend pool configuration and request logs answer it per request, which is also the answer to a data residency question about a specific region.
  • What safety controls were in effect, and did they fire? The content safety policy is versioned in the repository and its outcomes are logged per request.
  • Who could call what, and when did that change? Tenant configuration is Terraform, so the answer is a git history rather than a recollection.
  • Can a specific interaction be reconstructed? Only if content logging was on for that product, which is exactly why that decision is per product and written down rather than implicit.

The point worth taking away: the gateway is what makes these answerable at all. Five applications calling five model endpoints directly can produce five different answers to each question, and the effort to reconcile them is the compliance cost that the platform investment removed.

The scorecard

Part 1 set five targets. Honest scoring, including the one that did not land cleanly.

  • Every product owner sees their own token spend within a day. Met, with the accuracy classes from Part 3 stated on the report: exact for non-streamed and batch, estimated for streamed, with a published residual.
  • No application holds a long-lived model credential. Met. Applications hold an Entra ID client credential and a product subscription key; neither reaches a model endpoint, and revocation is a configuration change.
  • A retirement notice produces a query, not an investigation. Met, and it required three earlier decisions rather than a tool: pinned versions, aliases, and per-request logging of the resolved deployment.
  • One tenant cannot exhaust another's quota. Met for synchronous traffic through per-subscription counter keys. Batch is bounded by the control plane being the only submitter, which is a process control rather than an enforced one, and that gap should be named rather than smoothed over.
  • Gateway latency under 60 ms at the 95th percentile for non-cached calls. This is the one to measure rather than assume, and it is the number most sensitive to choices made for other reasons: the content safety check adds a call before the model, semantic cache lookup adds an embedding call, and both were adopted for good reasons that cost milliseconds. A platform that added all of them without re-measuring has quietly changed its own SLO.

What this cost, and when it is not worth it

The series would be dishonest without this section. A central gateway is a platform team, an API Management instance in a tier chosen for a schema rather than for load, a Redis cache with a module that cannot be added later, an embeddings deployment serving two purposes, and a control plane that is a real service with real on-call. That is a standing cost, and it buys attribution, governance, and a lifecycle story.

Below roughly three consuming applications, it is not worth it. The coordination cost the gateway removes does not exist yet, and the same discipline can be had with a shared Terraform module and a convention. The threshold is not a number of tokens, it is the point where a change to how models are accessed requires more than one team to act. That is when the second door starts costing more than the first one did.

Read this next

For the infrastructure and platform side of running this at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.

References