Caching strategies for LLM applications

Caching is one of the oldest cost-reduction techniques in software. In LLM applications, it should be one of the first things you reach for. The problem is that caching LLM responses is structurally different from caching database queries, and the naive approach creates a class of bug that is hard to detect: silently stale answers.

This post covers the three meaningful caching layers for LLM systems, when each applies, and what the failure modes look like.

Why LLM caching is different

A database cache key is usually exact: a user ID, a query string, a hash of the parameters. The response is deterministic. Stale means the underlying data changed.

LLM caches have two additional complications.

First, inputs are rarely exact matches. “What is the capital of France?” and “Can you tell me the capital of France?” are semantically identical but lexically different. Exact key caching misses both of them unless you normalize aggressively.

Second, staleness is semantic, not temporal. An LLM response about your product features goes stale when your features change, not after a fixed TTL. The response about a regulatory requirement goes stale when the regulation changes. There is no clock you can set that handles this correctly across all response types.

These two properties mean you need a more layered approach than a simple key-value cache.

Layer 1: Exact caching

The simplest layer. Hash the entire prompt (including system message, model, temperature, and any other parameters that affect output) and cache the response. Return it on exact match.

This is worth doing even if it hits rarely. Exact duplicates do occur in practice: onboarding flows where users see the same prompt, chatbots where many users ask identical FAQ questions, batch processing pipelines that reprocess overlapping data.

Exact caching is also the only layer that is safe for every response type. If the input is byte-for-byte identical and your system prompt has not changed, the cached response is valid.

Implementation is straightforward: Redis or Memcached with a composite key built from a hash of the full prompt context. TTL should be long enough to get value but short enough that it expires if your prompt templates change. Thirty days is a reasonable starting point; adjust based on how frequently your system prompts evolve.

Layer 2: Semantic caching

Semantic caching uses embeddings to find near-duplicate inputs. You embed each incoming prompt, query a vector store for similar prompts above a similarity threshold, and return the cached response if one exists.

This is where most of the savings come from in conversational applications. Users rephrase the same questions constantly. A similarity threshold around 0.95 cosine similarity catches rephrasing while avoiding false positives that would return wrong answers.

The risks are proportional to how strictly you need correctness. For factual queries about stable knowledge, semantic caching is safe. For queries about current state, user-specific context, or anything where small differences in phrasing might legitimately require different answers, the threshold needs to be higher or the cache needs to be bypassed entirely.

The failure mode to watch for: a 0.92 similarity cache hit returns an answer that was correct for a slightly different question. Users notice this as the system “not understanding” what they asked. It looks like a model quality problem, not a caching problem. Log cache hits and their similarity scores so you can audit them when users report unexpected answers.

Layer 3: Prompt prefix caching

Most LLM providers now support prompt prefix caching (also called context caching or KV cache reuse). If the beginning of your prompt is identical across requests, the provider processes it once and reuses the result.

This matters for applications with large system prompts: extensive instructions, long documents, retrieved knowledge base chunks. If your system prompt is 10,000 tokens and your user message is 100 tokens, prefix caching eliminates 99% of the input token processing cost for every request after the first.

Prefix caching is handled at the API level with minimal application changes. The main requirement is that the cacheable portion of your prompt comes first and stays stable. Avoid constructing system prompts dynamically on every request unless the dynamic portion is necessary.

What not to cache

Three categories should be excluded from all caching layers:

User-specific context. Any response that incorporates the user’s data, history, or state should not be shared across users. This is an obvious correctness requirement but easy to violate in semantic caching if user context is embedded in the query.

Current-state queries. Anything that needs to reflect live data (account balances, order status, real-time availability) must bypass the cache entirely. The cache key will match, the response will be wrong.

Critical decisions with audit requirements. If a response will be acted on in a consequential way and you need an audit trail, caching makes that trail incomplete. Each decision should be freshly generated and logged.

Managing staleness

The hardest problem in LLM caching is not building the cache, it is invalidating it correctly.

Exact and semantic caches need to be invalidated when your prompt templates change, when your underlying knowledge base changes, or when the facts the model was referencing have changed. None of these events map cleanly to a TTL.

Practical approaches: keep cache TTLs conservative (days, not weeks) for anything factual, tag cached entries with the version of your system prompt so you can bulk-invalidate on template updates, and treat semantic cache invalidation as a maintenance operation that runs on your knowledge base update cadence rather than a passive TTL.

Prefix caching at the provider level is typically handled with shorter TTLs (minutes to hours depending on the provider). The main risk is that you are billed at the full rate if the cache expires between requests. Batch similar requests together when possible.

What to instrument

Cache hit rate by layer. Cost per request before and after caching. Semantic similarity distribution of cache hits (to catch threshold tuning problems). Staleness events when you do find incorrect cached responses.

The return on a well-tuned LLM cache is significant: 30-70% cost reduction on repetitive query workloads, and latency improvements that users notice. The risk is invisible degradation that looks like model quality problems. Instrument both sides.

The right order to implement

Start with provider-level prefix caching: minimal code change, immediate cost reduction on large system prompts.

Add exact caching next: simple to implement correctly, zero staleness risk.

Add semantic caching only once you understand your query distribution well enough to set a reasonable similarity threshold and exclusion rules. It delivers the most savings but requires the most operational attention.