Microsoft's own policy documentation says a score threshold above 0.2 may lead to cache mismatch, which is a polite way of saying the gateway will answer a question the caller did not ask. Every other mechanism in this series has failed loudly: a 429, a 503, an expired batch job. A semantic cache fails by returning a fluent, well-formed, confident response that belongs to somebody else's prompt. Part 5 made spend attributable. This part makes it smaller, and treats the mechanism that does so as a correctness surface rather than a cost lever.

What the cache actually needs before it works

Semantic caching in API Management has a prerequisite list that decides an infrastructure choice you cannot revisit later, so it belongs before the policy rather than after it.

  • A second model deployment, for embeddings. The chat completion deployment serves consumer calls; a separate embeddings deployment is what the cache uses to turn a prompt into a vector. That is a second capacity line, a second quota, and a second thing to monitor.
  • Managed identity authentication from API Management to the model APIs, which the policy requires: embeddings-backend-auth must be set to system-assigned. There is no key-based option here, which is a good constraint.
  • Azure Managed Redis with the RediSearch module enabled, configured as an external cache on the API Management instance.

That last item carries the trap. RediSearch can only be enabled when the cache is created, and cannot be added to an existing cache. A platform that already runs a Redis instance for sessions or rate-limit state cannot simply turn semantic caching on: it needs a new cache, provisioned with the module, and the Terraform in Part 2 is the place that decision gets recorded. Discovering it during a sprint that budgeted an afternoon for "enable caching" is the ordinary outcome.

The policy pair

Lookup goes in inbound, store goes in outbound, and the pair can each appear only once per policy section. Both are available at global, product, API, and operation scope, and on classic, v2, consumption, and self-hosted gateways.

<inbound>
  <base />
  <llm-semantic-cache-lookup
      score-threshold="0.05"
      embeddings-backend-id="embeddings-backend"
      embeddings-backend-auth="system-assigned"
      ignore-system-messages="true"
      max-message-count="6">
    <vary-by>@(context.Subscription.Id)</vary-by>
  </llm-semantic-cache-lookup>
  <rate-limit calls="10" renewal-period="60" />
</inbound>
<outbound>
  <llm-semantic-cache-store duration="60" />
  <base />
</outbound>

Four of those attributes are decisions rather than boilerplate.

score-threshold defines how closely an incoming prompt must match a cached one, on a scale from 0.0 to 1.0, where lower values require higher semantic similarity. This inverts what most people assume on first reading, and getting the direction wrong turns a conservative cache into a promiscuous one. Microsoft's guidance is to start low, around 0.05, and tune toward the hit-to-miss ratio you want, with the explicit warning that above 0.2 mismatches become likely and that sensitive use cases should stay lower.

ignore-system-messages="true" strips system messages before similarity is assessed, and is recommended. It matters more than it sounds: two applications with identical user questions and different system prompts would otherwise look different to the cache, and a single application whose system prompt gets edited would invalidate its entire cache for no semantic reason.

max-message-count skips caching once a dialog has more than the specified number of remaining messages. Long conversations are exactly where a semantic match is least trustworthy, because the meaning of the last message depends on turns the embedding never saw.

vary-by is the isolation boundary, and the next section is about why it is a security control.

vary-by is not a partitioning nicety

Without vary-by, one cache is shared across every caller of the API. A prompt from the customer-service assistant containing a customer's details can be answered from a cached completion, and a semantically similar prompt from a different tenant can be answered from that same entry. The documentation puts it plainly: control cross-user access to cache entries by specifying vary-by with specific user or user-group identifiers.

For this platform the minimum is the subscription ID, which aligns the cache boundary exactly with the tenancy boundary Part 5 built. Where an application serves end users whose data must not mix, the correct value is a claim from the validated token rather than anything the caller can set in a header, which is the reason Part 5's validate-jwt writes its token into output-token-variable-name. A vary-by that reads an unvalidated header is a cross-tenant data leak with a configuration file for a root cause.

The cost of narrow partitioning is a lower hit rate, and that trade is real. It is also the correct default: a cache that never leaks and saves 20 percent beats one that saves 45 percent and eventually returns the wrong customer's answer.

The dependency you just added

The cache sits in the request path of every call, which means Redis is now on the critical path of the gateway. Microsoft's recommendation is specific and worth following exactly: place a rate-limit or rate-limit-by-key policy immediately after the cache lookup, to keep the backend from being overwhelmed if the cache is not available.

Think through that failure. On a normal day a meaningful share of traffic never reaches a model. If Redis becomes unavailable, every one of those requests becomes a real completion call, instantly, against a quota sized for the cached steady state. The cache outage does not degrade the platform, it multiplies its load, and the rate limit is what converts that from an outage into throttling. This is the same lesson as any cache in front of a database, and it is easier to forget here because the cache was introduced as a cost optimization rather than as a capacity dependency.

The embeddings deployment deserves the same thought. Every lookup embeds the incoming prompt, so the embeddings model needs enough capacity and enough context size for the prompt volume and prompt lengths in production. An embeddings deployment sized for a proof of concept becomes the bottleneck for all traffic, cached or not, because the lookup happens before the hit or miss is known.

What a cache hit does to Part 5's numbers

A hit is not free and it is not a completion. It costs one embeddings call, some Redis time, and no completion tokens at all, which means the chargeback model from Part 5 needs three small changes rather than a rewrite.

  • Embeddings spend becomes a platform cost line. It is incurred on every request including misses, and it is not attributable to a completion. Either it is charged to tenants pro rata by request count, or it is absorbed by the platform. Pick one and write it down; do not leave it out of the reconciliation.
  • Hit rate is a legitimate metric dimension. It is low cardinality and it belongs on the dashboard next to spend, because a hit rate that drops is usually a prompt template change nobody announced.
  • Savings should be reported as avoided cost, not as spend. A tenant whose bill fell 30 percent because of caching will assume the lower number is the new baseline. Showing avoided tokens alongside billed tokens keeps that conversation honest, and makes the case for the Redis instance the platform is paying for.

When not to cache at all

Semantic caching suits high-volume, low-variance, non-personalized prompts. The company's retail knowledge search over product documentation is close to ideal. The others are not, and the policy should be scoped per API rather than applied globally, which is exactly why it supports product and API scope.

Skip it where the response depends on the current time or on live state, because a 60 second duration is a correctness window, not just a freshness preference. Skip it for tool-calling flows, where the model's output is an instruction to act rather than text to read, and a stale instruction is a wrong action. Skip it for anything personalized beyond what vary-by can partition. And treat long multi-turn dialogs as out of scope by setting max-message-count rather than by hoping the similarity score notices.

Failure modes to watch

  • The threshold set backwards. Higher is looser. A team optimizing for hit rate raises it to 0.4, the ratio improves, and the mismatches are invisible until a user reports an answer that has nothing to do with their question.
  • A missing vary-by. No error, no warning, and a shared cache across every tenant on the API.
  • RediSearch discovered late. The module cannot be added to an existing cache, so the answer is a new cache and a migration, in the middle of the work that assumed a config change.
  • Redis down, backend flooded. Without the rate limit right after the lookup, a cache outage arrives at the model as a traffic spike.
  • An undersized embeddings deployment. It is called on every request, so it throttles hits and misses alike, and the symptom looks like the cache making things slower.
  • A prompt template change that silently empties the cache. Worth alerting on hit rate rather than discovering it on the bill.

What Part 7 inherits

The cache answers repeats. It does nothing for the far more common case where the model needs information it was never trained on, which is the retrieval problem, and the company's retail knowledge search is already waiting on it. Next: Azure AI Search against a dedicated vector database, compared on the terms that actually decide it.

Read this next

For the infrastructure and platform side of running this at scale, the field notes are at ercan.cloud, and the hub is at ercanermis.com.

References