How to Actually Cache an LLM App: Exact, Semantic, or Both

Let's face it: your LLM app answers the same questions all day, and you pay full price every time.

The user asks "What is your refund policy?" at 9:04. Another user asks "How do refunds work?" at 9:07. A third asks "What is your refund policy?" - character for character - at 9:11. That is three LLM calls, three multi-second waits, three line items on the invoice, for one answer that has not changed since last quarter.

Caching fixes this, but LLM caching has two failure modes, and they pull in opposite directions. Cache too little and you burn money re-computing identical work. Cache too eagerly and you serve the Berlin weather to someone who asked about Paris. Most of the "add a cache" advice out there ignores the second failure mode entirely, which is how semantic caches get a reputation for confidently returning wrong answers.

We ship two MIT-licensed libraries that sit on either side of that trade-off: @betterdb/agent-cache (exact match) and @betterdb/semantic-cache (similarity match), both with Python twins on PyPI. People keep asking which one they should use. The honest answer is: it depends on which of your requests are literally identical and which are merely equivalent - and most real apps have both. This post is the decision tree, with the numbers to back it.

Two caches, one decision

The two libraries are siblings, not layers. Neither imports the other. They share the same adapter set, the same observability stack, and the same discovery protocol, but they answer different questions:

	`@betterdb/agent-cache`	`@betterdb/semantic-cache`
Matching	Exact - SHA-256 of canonicalized request params	Approximate - embeddings + HNSW cosine KNN
Catches	Byte-identical requests	Paraphrases ("refund policy?" ≈ "how do refunds work?")
Server requirement	Vanilla Valkey 7+ / Redis 6.2+, no modules	Valkey 8+ with the `valkey-search` module
Runs on ElastiCache / Memorystore	Yes, any tier	Yes, on Valkey 8+ offerings with vector search
Scope	Three tiers: LLM responses, tool results, session state	One tier: prompt → response
Embedding calls	None	One per lookup (cached for 24h per unique text)
False positives	Impossible	Possible - this is the knob you tune
Wrong-answer risk	Zero	Managed via threshold, rerank, LLM judge

The exact cache can never be wrong: either the request hash matches or it does not. Its only weakness is that "What is your refund policy?" and "What's your refund policy?" are different SHA-256 hashes, so the paraphrase pays full price. The semantic cache exists precisely to catch that case - and in exchange you take on a precision/recall trade-off that you need to actually manage, not vibe.

That framing gives you the decision tree:

Tool results, session state, retried/replayed requests, temperature: 0 calls behind deterministic pipelines → exact. Free wins, zero risk.
Open-ended user questions where phrasing varies but answers do not (FAQ, support, documentation Q&A, RAG over a stable corpus) → semantic.
A real app with both kinds of traffic → both, as a waterfall. Exact first, semantic second, LLM last.

Start exact: the boring cache that always works

Start with agent-cache even if the semantic cache is what got you reading. It has no false positives, no threshold to tune, no embedding dependency, and it runs on any managed Redis-compatible service you already have - we wrote about why the no-modules constraint matters when the alternative libraries break on deploy day.

One connection, three tiers:

import Valkey from 'iovalkey';
import { AgentCache } from '@betterdb/agent-cache';

const client = new Valkey({ host: 'localhost', port: 6379 });
const cache = new AgentCache({
  client,
  tierDefaults: { llm: { ttl: 3600 }, tool: { ttl: 300 }, session: { ttl: 1800 } },
});

// LLM tier: keyed on model + messages + sampling params
const params = { model: 'gpt-4o-mini', messages, temperature: 0 };
const result = await cache.llm.check(params);
if (!result.hit) {
  const response = await callLlm(params);
  await cache.llm.store(params, response.text, {
    tokens: { input: response.usage.inputTokens, output: response.usage.outputTokens },
  });
}

// Tool tier: keyed on tool name + canonicalized args
const weather = await cache.tool.check('get_weather', { city: 'Sofia' });
if (!weather.hit) {
  await cache.tool.store('get_weather', { city: 'Sofia' }, JSON.stringify(data), { ttl: 300 });
}

// Session tier: per-thread state with sliding TTL
await cache.session.set('thread-1', 'last_intent', 'book_flight');

The tool tier is the one people underrate. An agent that calls get_weather({city: "Sofia"}) four times in one conversation - or across forty users' conversations - does not need four API calls. Tool results are the most cacheable thing in an agent workload because the arguments are structured, so "identical request" is actually common, unlike free-text prompts. Argument objects are canonicalized with recursive key sorting before hashing, so {city: "Sofia", units: "metric"} and {units: "metric", city: "Sofia"} are the same entry.

In our LangChain example, the cold call takes 1,032 ms and the cached call takes 1 ms. That is not a benchmark, it is a single illustrative run - but the shape is what matters: a cache hit is a Valkey GET, so it costs what a GET costs.

Three things to get right on day one:

Pin your sampling params. The cache key includes temperature, and omitting it is treated as the provider default of 1. If half your call sites pass temperature: 0 and half omit it, you have two cache populations that never hit each other. Pick one convention.
Set TTLs. The default is no expiry, and the library deliberately implements no eviction of its own - that is Valkey's job. Either set tierDefaults or configure a maxmemory policy, ideally both.
Attach token counts at store time. The library ships a price table covering 1,900+ models (generated from LiteLLM's pricing data), and if you pass tokens: {input, output} on store, every subsequent hit is credited in dollars in stats(). Skip it and your cost-savings number silently reads zero, which makes for a much less persuasive graph when someone asks whether the cache is worth keeping.

The same API exists in Python (pip install betterdb-agent-cache), fully async, with the same key format - a TS writer and a Python reader hitting the same Valkey produce the same hashes. Adapters cover LangChain, LangGraph checkpointing, the Vercel AI SDK (TS, including streaming), the OpenAI Agents SDK and Pydantic AI (Python), plus param normalizers for raw OpenAI, Anthropic, and LlamaIndex calls.

Go semantic when paraphrases are the workload

The semantic cache embeds each prompt, stores the vector in a Valkey HNSW index, and answers lookups with a cosine KNN search:

import Valkey from 'iovalkey';
import { SemanticCache } from '@betterdb/semantic-cache';
import { createOpenAIEmbed } from '@betterdb/semantic-cache/embed/openai';

const cache = new SemanticCache({
  client: new Valkey({ host: 'localhost', port: 6399 }),
  embedFn: createOpenAIEmbed(),   // or Voyage, Cohere, Bedrock, Ollama - any (text) => number[]
  defaultThreshold: 0.1,
  defaultTtl: 3600,
});
await cache.initialize();

await cache.store('What is the capital of France?', 'Paris', {
  model: 'gpt-4o-mini', inputTokens: 20, outputTokens: 5,
});

const hit = await cache.check('What city is the capital of France?');
// hit.hit === true, hit.similarity ≈ 0.087, hit.confidence === 'high' | 'uncertain'

The score is a distance, not a similarity - lower is closer. Exact wording scores ≈ 0.000, a close paraphrase lands around 0.08–0.09 with text-embedding-3-small, and a hit means score <= threshold. The result field is called similarity for familiarity, and yes, we have mixed feelings about that too.

The threshold is the entire game, and you do not need a table to start. Ship the default, 0.10 (roughly "0.90 similarity" in the convention other caches use), then move it with two rules:

Loosen - toward 0.15–0.20 - when real paraphrases are missing. The signal: misses keep landing just above the line. Every miss returns nearestMiss: { similarity, deltaToThreshold }, so you will see them stacking up at +0.02 instead of guessing.
Tighten - toward 0.05 - when wrong answers slip through. The signal: a growing share of hits classify as uncertain, or your judge keeps rejecting them.

One number rarely fits all traffic, which is what categoryThresholds is for - run faq tight at 0.08 and conversational loose at 0.15 in the same cache.

Two mechanisms keep the loose end honest. First, every hit is classified high or uncertain based on an uncertainty band below the threshold. Second, you can attach an LLM-as-judge that adjudicates only the uncertain hits - a cheap model gets the prompt and the cached response and answers "does this actually answer it?" A rerank hook (createKeywordOverlapRerank) additionally catches the classic embedding failure where "weather in Paris" and "weather in Berlin" sit close in vector space because the sentence structure is identical and only the entity differs.

Thresholds are also not transferable across embedding models - the score geometry belongs to the model, and the index dimension is locked at first initialize. Changing embedding models means flush() and re-tune. Budget for that before you start, not after you have a warm cache.

What the benchmarks actually say

We replay labeled paraphrase datasets through the real cache against a real Valkey - harness and methodology are in the open (we published the full series, including head-to-heads against RedisVL and Upstash). Here is one run that shows the trade-off curve: STS-Benchmark, 5,000 query pairs, bge-small-en-v1.5 embeddings computed locally, local Valkey with valkey-search:

Threshold	Hit rate	Precision	Recall	F1	p50 lookup
0.10	35.0%	0.713	0.472	0.568	5.7 ms
0.15	53.0%	0.713	0.715	0.714	5.7 ms
0.20	69.0%	0.674	0.879	0.763	5.6 ms
0.30	90.0%	0.578	0.984	0.728	5.8 ms
0.20 (judged)	56.1%	0.727	0.771	0.748	5.6 ms

Read the table honestly and three things fall out:

The lookup itself is essentially free. ~5–6 ms at p50, ~10 ms at p95, flat across thresholds. Against a 2,000 ms LLM call, the cache layer is noise. (For comparison, the same workload against a managed cloud vector cache ran at ~270 ms p50 - network round trips dominate, which is an argument for keeping the cache next to your app.)
Loosening the threshold buys recall with precision. At 0.20 you catch 88% of true paraphrases but one in three hits is wrong on this dataset. STSb is deliberately adversarial - it is full of near-miss pairs designed to fool similarity models - so treat these precision numbers as a floor, not a forecast for your FAQ traffic. But the direction of the curve is real, and it is why the default ships at a conservative 0.10.
The judge earns its keep at loose thresholds. At 0.20 it lifts precision from 0.674 to 0.727 and cuts the false-positive rate by a third, in exchange for hit rate and tail latency: p95 jumps to ~750 ms because uncertain hits wait on a gpt-4o-mini call. The p50 stays at 5.6 ms - most hits never see the judge. That is the correct shape for the trade: pay latency only on the borderline cases, where the alternative was serving a wrong answer.

Running both: the waterfall

Here is the part where the two libraries stop being an either/or. The pattern we run in production - the same one behind our RAG demo's two-tier setup - is a waterfall: exact first, semantic second, LLM last.

async function answer(messages: Message[], model = 'gpt-4o-mini') {
  const params = { model, messages, temperature: 0 };

  // Tier 1: exact. A Valkey GET - no embedding call, no false positives.
  const exact = await agentCache.llm.check(params);
  if (exact.hit) return exact.response;

  // Tier 2: semantic. One embedding + one KNN search on the last user message.
  const prompt = lastUserText(messages);
  const semantic = await semanticCache.check(prompt, { category: 'support' });
  if (semantic.hit && semantic.confidence === 'high') return semantic.response;

  // Tier 3: pay for the LLM call, then feed both caches.
  const res = await callLlm(params);
  await agentCache.llm.store(params, res.text, {
    tokens: { input: res.usage.inputTokens, output: res.usage.outputTokens },
  });
  await semanticCache.store(prompt, res.text, {
    model, inputTokens: res.usage.inputTokens, outputTokens: res.usage.outputTokens,
  });
  return res.text;
}

Why this ordering works:

The exact tier absorbs repeats for free. Retries, double-submits, identical questions from different users, replayed agent runs - all resolved by a GET before an embedding API is ever touched. Every request the exact tier absorbs is also one the semantic tier cannot get wrong.
The semantic tier only sees novel phrasings, which is exactly the traffic it is for. Gate it on confidence === 'high' (or attach the judge) and treat everything else as a miss.
Both tiers get fed on a miss, so the next identical request hits tier 1 and the next paraphrase hits tier 2.

And the division of labor outside the waterfall stays strict: tool results and session state live in agent-cache only. get_weather({city: "Paris"}) and get_weather({city: "Berlin"}) must never be "similar" - tool arguments are parameters, not prose. Semantic matching on structured data is how you get creative incident reports.

Tuning it in production

A threshold you picked from a blog post - including this one - is a starting point, not a setting. Both libraries assume you will tune them against live traffic, so they hand you the signals. This section is about reading them; the next one is about acting on them.

thresholdEffectiveness() watches your misses for you. It aggregates the last 10,000 lookups from a rolling window and recommends tighten_threshold, loosen_threshold, or optimal - the two adjustment rules from above, computed from your actual traffic instead of your intuition, with per-category breakdowns.
toolEffectiveness() does the same for tool caching - per-tool hit rates with TTL recommendations (a tool at 95% hit rate and a 60-second TTL is leaving money on the table; a tool at 12% is churning writes for nothing).
Every operation emits an OpenTelemetry span and Prometheus metrics - hit/miss counters, similarity histograms, cost-saved totals. Wire the spans into your existing tracing and a cache hit shows up in the same trace as the LLM span it replaced (here is what that looks like end to end).

The closed loop: retune a live cache without a redeploy

A static threshold is where most semantic caches stop. RedisVL, Upstash Vector, Redis LangCache - you pick a number at deploy time and you live with it until the next release. That is the part we think is wrong, and it is the main reason these libraries exist as more than a FT.SEARCH wrapper.

Both caches re-read their configuration from a hash on the Valkey server itself every 30 seconds. Thresholds, per-category thresholds, per-tool TTLs - all live-adjustable, no restart, no redeploy, every connected instance at once. So acting on a loosen_threshold recommendation from the last section is one write:

await client.hset('betterdb_scache:__config', 'threshold', '0.13');
// Every connected instance picks it up within 30 seconds. No redeploy.

The agent cache closes the same loop for tools: when the signal says increase_ttl for get_weather, cache.tool.setPolicy('get_weather', { ttl: 600 }) applies it fleet-wide through the same mechanism.

Run the loop by hand at first - read the recommendation, apply it, watch the hit rate. Then stop running it by hand. Each cache registers a discovery marker on the Valkey server (metadata only, never prompt data), so BetterDB Monitor finds running caches automatically, dashboards the signals from the last section, and writes its recommendations into this same config hash - the read, recommend, apply cycle running continuously against live traffic. We let it retune a cache autonomously for a week and published what happened. A cache with a feedback loop gets better after you ship it; a cache with a config file just gets stale.

What not to cache

A caching post that never says "don't" is an incident post waiting to happen. Skip the cache for:

Creative generation. If you sample at temperature: 1 because you want variety, caching defeats the point. The exact cache keys on temperature, which protects you from accidental collisions, but the real fix is not routing that traffic through a cache at all.
Tools with side effects. Caching get_weather is free money. Caching send_email is sending the email zero times and reporting success. Cache read-only tools; let everything else through.
Time-sensitive answers beyond their honesty window. "What's our current uptime?" cached for an hour is a lie fifty-nine minutes long. This is what per-tool TTL policies and short semantic TTLs are for.
Cross-user private data, carelessly. A semantic cache shared across users will happily serve user A's cached answer to user B's similar question. For personalized responses, scope the cache - per-user categories, filters, or separate namespaces - or keep personalized traffic out entirely.
Streams, in Python. The TS Vercel AI SDK middleware caches streaming responses (accumulate on miss, replay on hit); the Python adapters deliberately pass streams through uncached. Know which side of that line your stack is on.

The short version

Cache exact-match first: tool results, session state, repeated requests. It cannot be wrong, it runs on the vanilla Valkey or Redis you already have, and a hit costs what a GET costs. Add semantic caching when paraphrase traffic is real money, start at threshold 0.10, gate on confidence, and let thresholdEffectiveness() tell you where to go from there. Run them as a waterfall and keep structured data out of the semantic tier.

Both libraries are MIT-licensed, TypeScript and Python, on npm and PyPI:

npm install @betterdb/agent-cache iovalkey
npm install @betterdb/semantic-cache iovalkey

pip install betterdb-agent-cache
pip install betterdb-semantic-cache

Source for both is in the BetterDB monorepo, along with the benchmark harness that produced every number in this post - no cloud account required to reproduce any of them. (Both libraries emit anonymous usage analytics by default; BETTERDB_TELEMETRY=false turns it off.)

Want to see this in action? The waterfall from this post runs live at chat.betterdb.com, and the app itself is public at BetterDB-inc/playground-chat - read the wiring instead of taking my word for it.

If you are running an LLM cache in production - ours or anyone's - I would genuinely like to hear where the thresholds landed for you. And if you are paying full price for the same answer three times before lunch: now you know.