In Azure Pipelines, an approval that is not completed within its timeout marks the stage skipped, not failed. A skipped stage is green in most dashboards and in most notification templates, so a model promotion that nobody looked at can complete a pipeline run that reports success while the change never went out, or worse, while a preceding stage already did half of it. Part 8 put applications on top of the gateway. This part gets the gateway itself into production repeatably, and turns the model retirement email from Part 1 into a database query.

What is actually being deployed

Four artifacts, changing at three different rates, which is why they are three pipelines rather than one.

  • Platform infrastructure from Part 2's 10-platform: API Management, AKS, network, observability. Changes a few times a year, deploys with a human approval, and nobody is in a hurry.
  • Model deployments and gateway configuration, the 20-models layer plus policy XML and the alias map. Changes weekly. This is the pipeline that needs the eval gate.
  • Tenant configuration, the 30-tenants layer. Changes on onboarding, and is the one place a self-service pull request from a consumer team makes sense.
  • The control plane container on AKS. Ordinary application CI/CD, and the least interesting of the four precisely because it is ordinary.

Policy XML deserves its own note. It is executed on every request, so it is production code, and it reaches production through the same pipeline with the same review as Terraform. The portal is for reading, not for editing, and the way to make that stick is a drift check in the pipeline that fails when deployed policy differs from the repository.

The service connection, and the timer nobody sets

Pipelines authenticate to Azure with an Azure Resource Manager service connection, and the recommended configuration is workload identity federation, with either an app registration or a managed identity, which removes secrets and secret management from the picture entirely. For a platform whose entire Part 5 argument was about credential lifecycle, using a stored secret here would be difficult to defend.

Two operational details are worth putting in the runbook rather than discovering. Azure Pipelines automatically disables service connections that have not been used for 100 days, and a service connection administrator or Project Administrator has to re-enable them. A pipeline that only runs on quarterly infrastructure changes is exactly the pipeline that trips this, and the failure arrives at the worst time, when something urgent needs deploying. And a service connection converted from a secret to federation can be reverted for seven days; after that, a new secret has to be created manually, and connections you converted and then manually altered cannot be reverted through the tool at all.

The eval gate belongs to the environment, not the job

The instinct is to add an eval step at the end of the deploy job. That makes the eval a thing the pipeline does, which means the pipeline's author can reorder it, skip it with a condition, or move it after the deployment it was supposed to gate.

Azure Pipelines has the right primitive: checks on the environment or on the service connection the stage consumes. Before a stage runs, all checks on all resources it uses must be satisfied, and a single negative decision denies the stage. Configure the eval as an Invoke REST API check that calls the control plane's eval service, and the gate stops belonging to the pipeline definition and starts belonging to the environment.

The finality rules matter here. Decisions from approvals and most checks are final. Invoke Azure Function and Invoke REST API checks are the exception, they can be rerun, and if you set a non-zero time between evaluations, the check's decision becomes non-final and it will be re-evaluated. For an eval gate, that is a design choice rather than a detail: a non-final check that keeps re-running is right for "the eval must still be passing when this stage finally starts", and a final one is right for "this build passed its evals, ship it". Pick one, and know that the asynchronous form configured in the recommended way is final.

Add a human approval on top of the eval, on the production environment, with the option to restrict approvers from approving their own runs enabled. And set the timeout deliberately, because of the lead of this post: an approval that expires marks the stage skipped, and skipped reads as green. The mitigation is a short timeout, alerting on skipped stages specifically, and never treating a green pipeline as evidence that a deployment happened.

What the eval actually checks

A golden set of cases the platform's consumers care about, versioned in the repository next to the policy XML, running against the candidate configuration through the gateway rather than against a model endpoint. That last point is what makes it a platform eval rather than a model benchmark: it exercises the alias resolution, the policies, the content safety filter, and the retrieval path, all of which can break a response without the model changing at all.

Three thresholds, and the third is the one people skip. An aggregate quality score against the golden set. A latency budget, because a model that is marginally better and meaningfully slower is not an upgrade for an interactive assistant. And a regression list: specific cases that must not get worse, regardless of the average. A new model version that improves the mean while breaking the three prompts the customer-service team depends on is the normal shape of a bad promotion, and only the regression list catches it.

Promotion, canary, and rollback are all one mechanism

Because Part 3 put an alias in front of a backend pool, a model version change does not touch any application. It is a configuration change with three shapes:

  • Canary: add the new deployment to the alias's pool at a low weight in the same priority group. A small share of traffic goes to it, the metrics from Part 5 show latency and token differences per model alias, and nothing had to be redeployed.
  • Promotion: shift the weights, then remove the old deployment from the pool.
  • Rollback: the reverse weight change. Seconds, not a redeploy, which is the entire argument for the alias existing.

The one thing this does not do is make responses identical across the canary boundary. Two model versions serving the same alias produce different outputs for the same prompt, so anything that caches, compares, or replays responses needs to know which version answered. That is a log field, added at the same time as the canary, not afterwards.

Retirement, which is now a query

The retirement email from Part 1 cost two days of grepping repositories. It should now cost a query, because three earlier decisions made the answer knowable. Part 2 pinned model versions and set version_upgrade_option deliberately, so no deployment moves on its own. Part 3 put every application behind an alias, so the mapping from alias to deployment is data. Part 5 logged every request with its tenant and its resolved model, so the set of tenants that actually used a deployment in the last 30 days is a log query rather than an assumption.

The retirement runbook is then four steps that fit in a ticket: query which tenants used the deployment recently; stand up the replacement deployment and run the eval gate against it; canary the alias and watch the regression list; flip and remove. The applications never learn that any of it happened, which is the outcome the whole series was for.

Failure modes to watch

  • Skipped read as succeeded. An expired approval skips the stage. Alert on the skipped state explicitly, in the pipeline notification and in whatever dashboard the team actually looks at.
  • The eval as a job step. If it can be reordered or conditioned away, it is documentation. As an environment check it is a gate.
  • Bypass without visibility. Bypassing a check requires administrator permission on the resource and is recorded with who did it, which is good, and worth surfacing in a weekly review rather than leaving in the checks panel.
  • A service connection disabled at 100 days. The quarterly infrastructure pipeline discovers it during an incident.
  • Portal edits to policy. Without a drift check, a hotfix made in the portal survives until the next deploy silently reverts it, which is the worst possible timing.
  • An eval set nobody updates. A golden set that has not changed in a year is measuring last year's product.

What Part 10 inherits

A gateway that ships through pipelines, with evals as gates and model changes as configuration. Every mechanism in the series is now in place, and every one of them emits something. The last part is about what to do with that: traces, evals in production rather than in the pipeline, the governance questions the EU AI Act asks, and the numbers Part 1 promised.

Read this next

For the infrastructure and platform side of running this at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.

References