A semantic cache answers "how do I reset my password" with the response it already generated for "I forgot my login," because the two mean the same thing. Instead of matching the exact string of a request, it embeds the request and looks for a stored request whose embedding is close enough, then returns that cached answer without calling the model. When it works, you skip a model call entirely: no tokens, no latency, no cost. When it misjudges "close enough," it serves a confident answer to a question the user did not ask.

The core reframing: a normal cache keys on identity, a semantic cache keys on similarity, and similarity is a threshold you set, not a fact. That threshold is the entire risk surface. Set it loose and you serve wrong answers; set it tight and you cache almost nothing. Everything about running a semantic cache well is managing that dial.

How it works, minus the magic

The mechanism is three steps. Embed the incoming request into a vector using an embeddings model such as Amazon Titan Text Embeddings or Cohere on Bedrock. Search a vector store for the nearest previously seen request. If the nearest neighbor's cosine similarity is above your threshold, return its cached response; otherwise call the model, then store the new request embedding and response for next time.

vec = embed(request)
hit = vector_store.nearest(vec)
if hit and hit.similarity >= THRESHOLD:
    return hit.cached_response      # no model call
answer = model.invoke(request)
vector_store.put(vec, answer)
return answer

The vector store can be anything you already run: OpenSearch, Aurora with pgvector, or an in-memory index for a small hot set. The hard part is never the storage. It is the value of THRESHOLD and what you allow into the cache in the first place.

The false-hit risk is the whole game

A false hit is when two requests are close in embedding space but should have different answers. "What is our refund policy for EU customers" and "what is our refund policy for US customers" can sit dangerously near each other in vector space while demanding different responses. A loose threshold returns the EU answer to a US user and does it with total confidence, because the model was never consulted.

This is worse than a normal cache miss, which just costs you a model call. A false hit costs you a wrong answer that looks right. So the threshold has to be tuned against real traffic, not guessed, and it should be biased toward tightness: a miss is cheap, a false hit can be a support ticket or a compliance problem. Measure the false-hit rate on a labeled sample before you trust the cache with anything that matters.

What not to cache

Semantic caching fits stable, general, shared knowledge and fits nothing else. Cache the answer to "how do I export my data," which is the same for everyone. Do not cache anything that depends on who is asking or when. Rules of thumb worth enforcing in code:

  • Never cache across identity boundaries. Key the cache per tenant, or exclude anything personalized. A shared cache that ignores the caller will eventually serve one customer's answer to another.
  • Never cache time-sensitive answers. Balances, order status, and inventory change under you. A stale hit here is a fresh bug.
  • Cache the retrieval-free, not the retrieval-heavy. If the answer depends on documents that change, cache the embedding lookup, not the final response.

Invalidation is the part everyone skips

The oldest joke in computing is that cache invalidation is one of the two hard problems, and a semantic cache does not get an exemption. When the underlying truth changes, when a policy is updated, a price moves, a document is revised, every cached answer that leaned on the old truth is now wrong and will keep being served until you evict it.

Give entries a time-to-live short enough that stale answers age out on their own, and add an explicit purge path tied to the events that change ground truth: publishing a new policy version should invalidate the slice of cache built on the old one. Without an invalidation story, a semantic cache does not just save calls, it quietly pins your product to yesterday's facts.

The takeaway

Semantic caching is a real win on cost and latency for questions that are the same question in different words, and it is a footgun for anything personalized, time-sensitive, or subject to change. The mechanism is trivial: embed, search, threshold. The engineering is entirely in the threshold, the scope rules, and the invalidation path. Tune the threshold against labeled traffic, cache only stable shared knowledge, and invalidate on the events that move the truth. Do those three things and the cache pays for itself. Skip them and it serves wrong answers faster than the model could have served right ones.

Read this next

For the platform side of running a vector store and its eviction under load, the cloud field notes live at ercan.cloud, and the hub is at ercanermis.com.