Semantic Caching

Semantic caching is a technique for large language model (LLM) applications in which previous model responses are stored together with embeddings of the prompts that produced them, so that a new request can be answered from the cache whenever it is close enough in meaning to an earlier one — rather than only when the wording matches exactly^[1].

Background

A conventional cache returns a stored value only when the key is identical, which rarely happens in chat: "what are your opening hours", "when do you open" and "buka pukul berapa" are three keys for one question. Vector search made meaning-addressable lookup cheap, and researchers have begun optimising the pattern directly — Semantic Caching for Low-Cost LLM Serving (2025) studies when cached inference responses can be safely reused without running the model again^[2].

Semantic caching is distinct from prompt caching, the provider-side feature that discounts or accelerates an identical prompt by reusing its computed key-value prefix. Semantic caching sits at the application layer, reuses whole answers, and works across differently worded prompts.

How it works

The incoming prompt is turned into an embedding by a small, fast embedding model.

A vector search looks for the nearest previously cached prompt embeddings.

If similarity is above a threshold — commonly around 0.90–0.95 cosine similarity — the stored response is returned immediately; otherwise the LLM is called and the new pair is stored.

Cache entries can be scoped by metadata such as model version, language or user segment to prevent inappropriate reuse.

The design is a trade-off: a threshold set too low produces false hits — an answer that looks relevant to a similar question but answers a subtly different one — while a threshold set too high yields few hits. On a hit, a database lookup measured in milliseconds replaces a model call measured in hundreds of milliseconds.

Benefits and risks

For workloads with repetitive questions — FAQs, catalogue lookups, status enquiries — vendor benchmarks report cost reductions of up to about 86% and latency improvements of around 88%^[3], because repeated queries cost nothing after the first. Risks include staleness (a cached price or policy outlives its source), over-broad hits across different users or contexts, and the fact that cached prompts and responses usually contain personal data. Teams therefore apply TTLs, version keys and redaction to the cache, not just to the live call.

>See Also

• Prompt Caching
• Vector Database
• Embedding
• Inference
• LLM Routing