Azure's batch inference path does not look like an API call at all. You upload a JSONL file, create a job, and collect an output file up to 24 hours later, which means there is no per-request transaction for a gateway policy to sit in front of. Every guarantee Part 3 built, token limits, per-tenant metering, routing, circuit breaking, applies to requests through API Management. Batch has none of them by construction. This part moves the work that should never have been on the request path in the first place, and deals with the fact that doing so opens a second door into the models.

What belongs off the request path

The company's five applications produce three kinds of work that are on the synchronous path only because that was the easiest place to put them. The test is not "is it slow", it is who is waiting.

  • Nightly summarization of support tickets. Tens of thousands of items, no human waiting, results needed by the morning report. A perfect batch candidate that currently competes for the same TPM quota as a customer mid-conversation.
  • Document ingestion for the retail knowledge search. Bursty, triggered by uploads, tolerant of minutes. A queue, not a batch job: latency in minutes matters, latency in hours does not work because someone uploaded a document expecting to find it.
  • Re-scoring after a prompt change. Runs against a corpus, no user at all, and gets cancelled halfway more often than it completes. Batch, with an explicit cancellation story.

Three shapes, and they map onto exactly three mechanisms: a queue with workers for minutes, a batch job for hours, and the synchronous gateway for everything a human is watching. Conflating them is how a marketing job takes down a customer-service assistant, which is the incident from Part 1.

The queue, and what not to put in it

Azure Service Bus carries the minutes-scale work. The design decisions that matter are all about what the message contains rather than which broker it is.

Do not put the prompt in the message. Service Bus caps each individual message property at 32 KB and the cumulative header, user properties plus system properties, at 64 KB, and exceeding it raises a serialization exception rather than truncating quietly. Even inside the body, a payload over 1 MB is counted twice against the entity's size quota. The claim-check pattern is the answer: the document goes to Blob Storage, the message carries a blob reference, a tenant ID, a model alias, and a correlation ID. The message stays small, the queue depth stays a meaningful metric, and the payload is already where the worker wants to stream it from.

Two more constraints shape the worker pool. A single queue, topic, or subscription accepts 5,000 concurrent receive requests before rejecting further receives with a server busy error, which is a ceiling on receiver count rather than on throughput, and it is reached faster than people expect by a worker that opens a receiver per task. And a namespace allows 5,000 concurrent AMQP connections, so connection pooling in the worker is not an optimization, it is a requirement at scale.

Dead-lettering is where an LLM queue differs from an ordinary one. A message that fails because the model returned a content filter block is not the same as one that failed because the worker crashed, and only the second should be retried. The worker completes the message and records the refusal as a result when the model answers with a refusal, and abandons it only for infrastructure failures. Otherwise the delivery count climbs to its limit on a message that will never succeed, and the dead-letter queue fills with items nobody can distinguish from real failures.

Scaling the workers with KEDA

The workers live on the AKS cluster Part 2 provisioned, and they should not run when the queue is empty. KEDA is available as an AKS add-on and scales workloads to zero, driving ScaledObject for deployments and ScaledJob for job-shaped work, with authentication decoupled from the workload through Microsoft Entra Workload ID.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: ingest-worker
spec:
  scaleTargetRef:
    name: ingest-worker
  minReplicaCount: 0
  maxReplicaCount: 20
  pollingInterval: 15
  cooldownPeriod: 120
  triggers:
    - type: azure-servicebus
      metadata:
        queueName: doc-ingest
        messageCount: "20"      # target backlog per replica
      authenticationRef:
        name: keda-workload-identity

Three add-on limitations decide whether this works on the first attempt or the third. Do not pair a ScaledObject with a Horizontal Pod Autoscaler on the same workload. KEDA uses an HPA underneath, so the two compete: if the HPA exists first, the ScaledObject fails to be created, and if the ScaledObject exists first, the HPA is created anyway and the scaling behaviour turns strange. Only one external metric server is allowed per cluster, so the KEDA add-on must be the only one and multiple KEDA installations are unsupported, which rules out a team installing its own via Helm alongside the platform's. And on AKS Standard, enable workload identity before enabling the KEDA add-on; done in the wrong order, the KEDA operator pods need a restart to pick up the right environment.

A useful default: scale on backlog per replica rather than on absolute queue depth, and set maxReplicaCount from the model quota rather than from the cluster's capacity. Twenty workers that each get a 429 are worse than five that do not, and the queue does not care how long a message waits.

The batch path, and why it is a second door

Hours-scale work goes to a Global-Batch model deployment, and the shape of that API is the point of this part. There is no request to intercept. A file is uploaded, a job is created, and an output file appears when the job completes. The mechanics are specific enough to be worth stating precisely.

The input is JSONL, one request object per line, and every line carries a custom_id:

{"custom_id": "ticket-88412", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "batch-summarize", "messages": [{"role": "system", "content": "Summarize the ticket in two sentences."}, {"role": "user", "content": "..."}]}}
{"custom_id": "ticket-88413", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "batch-summarize", "messages": [{"role": "system", "content": "Summarize the ticket in two sentences."}, {"role": "user", "content": "..."}]}}

Responses are not returned in the order the file defined, which is why custom_id is required rather than convenient: it is the only way to join a response back to its input. The model attribute must name the Global Batch deployment, and the same deployment name must appear on every line. Targeting a second deployment means a second file and a second job, which makes "route this batch to whichever model is cheaper today" a submission-time decision rather than a routing one. Microsoft's own guidance is to submit large files rather than many small ones.

batch = client.batches.create(
    input_file_id=file_id,
    endpoint="/chat/completions",
    completion_window="24h",
    # 1209600 to 2592000 seconds, 14 to 30 days, before the output file expires
    extra_body={"output_expires_after": {"seconds": 1209600, "anchor": "created_at"}},
)

The job then moves through validating, in_progress, finalizing, and completed, and carries an expires_at 24 hours after creation along with a running request_counts of completed, failed, and total. The completion window is 24 hours, and a job that does not finish inside it expires rather than continuing. Output files themselves expire too, on a window that can be set between 14 and 30 days, so a pipeline that assumes results are still there next quarter is a data loss waiting to happen.

Capacity works differently here as well. Batch jobs consume an enqueued token quota, and a job large enough to exceed it is rejected rather than queued behind the previous one. Certain regions now support a fail-fast behaviour that lets several batch jobs be queued with exponential backoff, so one finishing kicks off the next automatically. Without that, the retry loop is the submitter's job, and it belongs in the control plane rather than in each application.

Keeping the accounting honest

Here is the uncomfortable part. The batch path does not traverse API Management, so llm-token-limit does not throttle it, llm-emit-token-metric does not meter it, and the per-tenant attribution Part 5 is about to build has a blind spot the size of the largest workload in the company.

Two answers, and the difference between them is worth deciding deliberately rather than by default.

  • Let applications submit batch jobs directly and accept a second, unmetered door. Simplest, and it quietly reintroduces the exact problem Part 1 was written about, one bill nobody can attribute.
  • Make the control plane the only batch submitter. An application posts a batch request to the control plane, which validates the tenant, resolves the model alias to a Global Batch deployment, writes the JSONL, submits the job, records the submission against the tenant, polls to completion, and hands back the output. The gateway is still the only door for synchronous traffic, and the control plane is the only door for asynchronous traffic.

The second is more work and it is the one that keeps the series' premise true. It also lands the accounting on firmer ground than the streaming case from Part 3: a completed batch job reports its own request_counts and the output file carries per-response usage, so batch spend is exactly attributable, more so than the streamed traffic on the synchronous path. That is a pleasant inversion worth knowing about before someone assumes async means approximate.

Failure modes to watch

  • The retry loop that cannot succeed. A content filter refusal is a result, not a failure. Retrying it burns quota, inflates the delivery count, and ends in a dead-letter queue full of messages that were answered correctly.
  • Workers scaled past the quota. KEDA happily scales to maxReplicaCount on backlog alone. If that exceeds what the model deployment's TPM allows, the extra replicas generate 429s and the queue drains no faster.
  • A batch job that expires quietly. The 24 hour window ends in an expired status, not an exception in anyone's code. Without an alert on job status, the morning report is simply missing and the first person to notice is the person who reads it.
  • Output files that age out. A results file has an expiry between 14 and 30 days. Anything that needs to be kept gets copied to storage the platform owns, on submission, not later.
  • The second door reopened by convenience. One team with direct access to the Global Batch deployment undoes the attribution model. The network rule and the alert from Part 1 apply here too, and this is the path they are most likely to miss.

What Part 5 inherits

Synchronous traffic through the gateway, minutes-scale work on a queue with workers that scale to zero, and hours-scale work submitted by the control plane against a Global Batch deployment. Three paths, three latency profiles, and three different qualities of usage data feeding the same question: which team spent what. That question is next, along with the identity model that makes "which team" answerable at all.

Read this next

For the infrastructure and platform side of running this at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.

References