Semantic Caching: How to Cut LLM API Costs by 80%

Output tokens on flagship models cost significantly more than input tokens, yet engineering teams routinely waste over 40% of their monthly AI spend on repeat queries that ask for the exact same information in slightly different ways. Traditional exact-match key-value caches like standard Redis string lookup fail here completely—if a user submits 'How do I cancel my plan?' and another submits 'Steps to cancel subscription?', exact string hashing treats them as two unique requests. Both hit your Large Language Model (LLM) API, costing you real money and adding 1,500 milliseconds of unnecessary latency.

Semantic caching solves this architectural inefficiency by caching responses based on intent rather than exact syntax. By converting incoming prompts into mathematical vector embeddings and running a similarity check against previously stored prompts, your application can intercept incoming requests at the gateway level and serve instant responses in under 20 milliseconds—cutting API costs by 60% to 80% on high-concurrency production workloads.

In this practical guide, we will step through the exact architecture, code implementation, threshold tuning, and invalidation strategies required to set up production-ready semantic caching for your LLM stack.

What Is Semantic Caching and How Does It Work?

Semantic caching is an intelligent data retrieval layer that sits between your frontend application and your upstream LLM provider (such as OpenAI, Anthropic, or self-hosted vLLM). Instead of indexing cached items using a plain text string or MD5 hash as the lookup key, a semantic cache converts the prompt text into an embedding vector—a dense numerical array representing its underlying context and meaning.

When a new request enters your API gateway or backend server, the workflow proceeds through five discrete operations:

1. Embedding Generation: The application passes the incoming prompt through a lightweight, cost-effective embedding model (like `text-embedding-3-small` or an open-source `bge-small-en-v1.5`).
2. Vector Similarity Search: The resulting vector is queried against an in-memory vector index containing previously cached prompt vectors.
3. Distance Evaluation: The cache measures the spatial closeness—typically via Cosine Similarity or Euclidean Distance—between the incoming query and the nearest neighbor in the database.
4. Threshold Check: If the similarity score meets or exceeds a predefined confidence threshold (e.g., 0.90), the system registers a Cache Hit and instantly returns the associated, pre-generated completion.
5. Upstream Call and Cache Writes: If the score falls below the threshold, the system registers a Cache Miss, routes the prompt to the upstream LLM API, delivers the generated response back to the user, and asynchronously stores the new prompt vector alongside its completion in the cache.

Exact-Match Caching vs. Semantic Caching

To understand why semantic caching is essential for AI applications, consider how traditional infrastructure handles input variability:

Step-by-Step Architecture for a Production Semantic Cache

Building a reliable semantic cache requires three core building blocks: an embedding engine, a fast vector store with scalar metadata support, and an orchestration module.

Architectural Flow

When an end user fires a request in your application:

1. The user's prompt arrives at your application backend or middleware proxy.
2. The system calls an embedding service (or local model) to generate an array of numbers representing the text.
3. The backend sends a vector search query to your vector store (e.g., Redis VL, Qdrant, or Pinecone).
4. The vector store identifies the closest match and calculates the similarity score.
5. If similarity is greater than or equal to `threshold`, the cache payload is returned immediately.
6. If similarity is less than `threshold`, the application invokes the LLM API, receives the response, stores the pair in the vector store, and returns the LLM response.

Python Implementation: Building a Custom Semantic Cache with Redis

While high-level frameworks like LangChain offer `RedisSemanticCache` out of the box, building an explicit pipeline in Python gives you full control over threshold fine-tuning, latency optimization, and metadata scoping.

Below is a production-grade implementation using Redis as the vector database alongside OpenAI's `text-embedding-3-small` model.

Prerequisites

First, install the necessary dependencies in your Python environment:

`pip install redis openai numpy`

Ensure you have a Redis instance running with the RediSearch module enabled (or run `docker run -d -p 6379:6379 redis/redis-stack:latest`).

Python Implementation Code

```python import os import json import numpy as np from openai import OpenAI import redis from redis.commands.search.field import VectorField, TextField from redis.commands.search.indexDefinition import IndexDefinition, IndexType from redis.commands.search.query import Query

class ProductionSemanticCache: def __init__( self, redis_host="localhost", redis_port=6379, index_name="llm_cache_idx", embedding_dim=1536, similarity_threshold=0.88 ): self.client = OpenAI(api_key=os.getenv("OPENAI_API_KEY")) self.r = redis.Redis(host=redis_host, port=redis_port, decode_responses=False) self.index_name = index_name self.embedding_dim = embedding_dim self.threshold = similarity_threshold

self._ensure_index_exists()

def _ensure_index_exists(self): """Creates an HNSW vector index in Redis if it does not already exist.""" try: self.r.ft(self.index_name).info() except redis.exceptions.ResponseError:

Define schema for vector index

schema = ( TextField("prompt"), TextField("response"), VectorField( "prompt_vector", "HNSW", { "TYPE": "FLOAT32", "DIM": self.embedding_dim, "DISTANCE_METRIC": "COSINE", "INITIAL_CAP": 10000, } ) ) definition = IndexDefinition(prefix=["cache:"], index_type=IndexType.HASH) self.r.ft(self.index_name).create_index(fields=schema, definition=definition)

def _get_embedding(self, text: str) -> list: """Generates a normalized embedding vector for the prompt.""" response = self.client.embeddings.create( model="text-embedding-3-small", input=text ) return response.data[0].embedding

def check_cache(self, prompt: str): """Searches Redis for a semantically similar prompt above the threshold.""" query_vector = self._get_embedding(prompt) vector_bytes = np.array(query_vector, dtype=np.float32).tobytes()

Query to fetch the nearest neighbor and calculate cosine distance

redis_query = ( Query("*=>[KNN 1 @prompt_vector $vec AS score]") .return_fields("prompt", "response", "score") .sort_by("score") .dialect(2) )

query_params = {"vec": vector_bytes} results = self.r.ft(self.index_name).search(redis_query, query_params=query_params)

if results.docs: top_doc = results.docs[0]

Convert Cosine Distance to Cosine Similarity score

cosine_distance = float(top_doc.score) similarity_score = 1.0 - cosine_distance

if similarity_score >= self.threshold: return { "hit": True, "similarity": similarity_score, "response": top_doc.response.decode('utf-8') if isinstance(top_doc.response, bytes) else top_doc.response }

return {"hit": False, "similarity": 0.0, "response": None}

def store_cache(self, prompt: str, response: str): """Stores a new prompt, its vector embedding, and the generated response.""" query_vector = self._get_embedding(prompt) vector_bytes = np.array(query_vector, dtype=np.float32).tobytes()

doc_id = f"cache:{hash(prompt)}" self.r.hset( doc_id, mapping={ "prompt": prompt, "response": response, "prompt_vector": vector_bytes } )

Example Usage Pipeline

if __name__ == "__main__": cache = ProductionSemanticCache(similarity_threshold=0.88)

user_prompt = "What are the rules for returning a defective product?"

1. Lookup in Cache

cache_result = cache.check_cache(user_prompt)

if cache_result["hit"]: print(f"[CACHE HIT - Similarity: {cache_result['similarity']:.4f}]") print(f"Response: {cache_result['response']}") else: print("[CACHE MISS] Querying Upstream LLM...")

Simulate LLM Response from OpenAI / Anthropic

llm_response = "Defective items can be returned within 30 days of purchase with full proof of receipt."

Save result to cache

cache.store_cache(user_prompt, llm_response) print(f"Response: {llm_response}") ```

How to Choose the Right Similarity Threshold

The single most critical tuning factor in any semantic cache is the similarity threshold. Set it too high, and your hit rate drops to zero, defeating the purpose of the cache. Set it too low, and your application suffers from hallucinatory cache hits—returning answers that answered a completely different underlying intent.

Cosine Similarity Ranges for Production Workloads

• 0.95 to 0.99 (Ultra-Conservative): Requires almost identical phrasing. Best for legal, medical, or financial AI assistants where small variations in terms change the compliance outcome entirely.
• 0.88 to 0.93 (Balanced / Recommended): The sweet spot for customer support, SaaS documentation bots, and internal search engines. Accommodates syntax swaps, minor typos, and structural variations without conflating meaning.
• 0.80 to 0.85 (Aggressive): Yields massive token savings but carries a high risk of false positives. Only use this level for general broad-topic summarizers or intent classification workflows where extreme precision is not mandatory.

Practical Threshold Testing Matrix

To find your optimal threshold, test 100 sample user query pairs across various categories using this baseline reference:

Evaluating Semantic Caching Frameworks and Infrastructure

When moving beyond a raw script into production infrastructure, you can choose between managed vector databases, open-source caching libraries, and dedicated gateway proxies.

1. Dedicated Vector Databases (Redis vs. Qdrant vs. Pinecone)

• Redis / Redis Cloud: The industry standard for low-latency semantic caching. Storing vector indexes in RAM alongside your existing sessions and key-value stores minimizes cross-network hops, delivering sub-15ms lookups.
• Qdrant: An exceptionally fast, Rust-based open-source vector database. Highly recommended if you manage large payload filtering or need on-premises deployment versatility.
• Pinecone: Fully managed serverless vector database. Extremely low maintenance, though API-based network overhead can add 30–60ms of latency per cache check compared to in-memory Redis instances.

2. High-Level Libraries (GPTCache vs. LangChain / LlamaIndex)

• GPTCache: An open-source, highly modular Python library dedicated specifically to semantic caching for LLMs. Supports multi-stage storage, pre-embedding filters, and custom distance metrics.
• LangChain / LlamaIndex Native Integrations: Easy to turn on with a couple of lines of code. Excellent for fast prototyping, but can be harder to customize when you need complex cache-busting, tenant isolation, or customized data sanitization.

3. API Gateway Level Caching (Portkey, LiteLLM, Gravitee)

Implementing semantic caching at the API Gateway level abstracts the cache completely away from your application code. Proxies like Portkey, LiteLLM, or Gravitee intercept outgoing HTTP requests to OpenAI/Anthropic, handle vector lookups automatically, and return the response before the payload ever reaches external networks.

Advanced Strategies: Handling Dynamic Data, Cache Invalidation, and Security

Deploying a semantic cache in production introduces real-world complexities that simple tutorials overlook. Here is how to handle edge cases safely.

1. Dynamic Variables and Context Injection

If your prompts contain dynamic user context—such as user names, organization IDs, dates, or account balances—a naive semantic search will treat prompts from two different users as identical:

• User A: "What is the account balance for Jane Doe?"
• User B: "What is the account balance for John Smith?"

Because the core intent is 95% identical, an unpartitioned semantic cache will serve Jane's balance to John—creating a critical security breach.

The Fix: Always separate system templates from variable payloads, or enforce hard metadata partitioning. When querying Redis or Qdrant, construct hybrid queries that filter strictly by `tenant_id` or `user_role` before executing the vector distance calculation.

```python

Filtered vector search example in Redis

query_str = "(@tenant_id:{acme_corp})=>[KNN 1 @prompt_vector $vec AS score]" ```

2. Time-To-Live (TTL) and Cache Invalidation

LLM knowledge bases change over time. If your RAG application indexes documentation updated daily, a perpetual semantic cache will serve stale, outdated data.

• Time-Based TTL: Attach an explicit TTL (e.g., 86,400 seconds for 24 hours) to all cache keys in your vector store so outdated entries expire naturally.
• Event-Driven Flushing: Hook your documentation update pipeline (CI/CD or database triggers) to invalidate specific vector namespaces whenever underlying source files change.

3. Asynchronous Cache Writes

Generating embeddings and writing to your vector database takes time (50–150 ms). Do not block the user's stream to update the cache. Use a background worker (e.g., Celery, Inngest, or FastAPI BackgroundTasks) to process vector generation and store operations asynchronously after the first chunk of the LLM stream is dispatched to the client.

Measuring ROI: Financial and Latency Impact

To understand the financial return on implementing a semantic cache, consider a mid-sized SaaS platform processing 1,000,000 LLM requests per month using a model like Claude 3.5 Sonnet or GPT-4o.

Cost & Speed Benchmark (Before vs. After)

• Average Prompt Size: 800 input tokens, 400 output tokens.
• Blended API Cost without Cache: ~$4,500 / month.
• Average API Response Latency: 1,400 ms.

Assuming a conservative 35% Semantic Cache Hit Rate achieved at a 0.89 similarity threshold:

• Bypassed API Calls: 350,000 requests / month.
• Direct Monthly API Cost Savings: ~$1,575 / month ($18,900 / year).
• Average Cache Hit Latency: 18 ms (a 98.7% reduction in response time for cached queries).
• Infrastructure Overhead: ~$40 / month for a managed Redis memory cluster.

Beyond direct cost reduction, dropping P95 response latency from 1.5 seconds down to under 20 milliseconds creates a significantly snappier user experience, especially in real-time conversational agents and web applications.

Key Takeaways

1. Focus on Intent, Not Syntax: Semantic caching saves tokens by matching incoming prompts based on vector similarity rather than exact string equality.
2. Tune Similarity Thresholds Wisely: Start with a cosine similarity threshold of 0.88 to 0.92 to prevent hallucinated cache hits while maintaining solid hit rates.
3. Partition by Tenant: Prevent data leakage by embedding metadata filters (`tenant_id`, `user_id`) directly into your vector search queries.
4. Decouple Cache Writes: Run embedding creation and cache stores asynchronously to keep end-user response times lightning fast.

Building cost-effective, scalable AI tools requires picking the right software stack early. At Saasbonus, we publish independent, hands-on reviews and architectural deep dives across vector databases, orchestration frameworks, and AI developer platforms—helping you make the right software choices the first time.