Caching for ML Systems: Tiers, Eviction, Semantic and KV Caches

“In the world of high-scale machine learning, the fastest inference is the one you never had to compute. Caching is not just about saving time; it’s about making the impossible latency targets possible.”

TL;DR

ML caching goes far beyond “put it in Redis.” Production systems use a three-tier hierarchy: in-process caches (nanosecond latency) for hot keys, distributed Redis clusters (1-5ms) for user features, and persistent SSD stores (5-20ms) for historical data. Semantic caching with vector similarity can reuse LLM generations for paraphrased prompts, and the savings depend on how repetitive your traffic is. Consistent hashing with virtual nodes prevents cache miss storms during node failures, and KV cache management via PagedAttention largely decides how many concurrent sequences fit on one GPU. For more on scaling the inference layer that caching protects, see the inference optimization techniques and the chatbot system design.

1. Introduction: The Inference Bottleneck

As machine learning models grow from simple Decision Trees to 70B parameter Transformers, the “Cost per Inference” has become a defining business metric.

• A single LLM generate call can cost cents.
• A real-time ranking call for 10,000 items can take hundreds of milliseconds.
• Storing pre-computed features for 1 billion users often requires Petabytes of storage.

If you are designing the Recommendation System for TikTok or the Search Ranking for Amazon, you cannot afford to call the heavy Re-ranking Model for every single user interaction. You must use Advanced Caching Strategies.

This post moves beyond the basics of “put it in Redis.” We explore the Multi-Tiered Cache Architecture, Semantic Caching for LLMs, Feature Store Hydration, and how to solve the dreaded Thundering Herd problem in distributed systems.

---

2. Functional Requirements of an ML Cache

ML Caching is distinct from standard Web Caching (e.g., caching HTML pages).

1. High Throughput, Low Latency: We need to fetch 1,000 features for a candidate batch in < 5ms.
2. Approximate Correctness: Unlike a Bank Ledger, it’s okay if a user’s “Interest Vector” is 5 minutes stale.
3. High Dimensionality: We aren’t just caching strings; we are caching Vectors (Embeddings) and Tensors.
4. Semantic Retrieval: Use Vector Similarity to find “near-matches” for user queries, allowing us to reuse LLM generations for similar prompts.

---

3. The Architecture: The Latency Hierarchy

Large ML serving stacks typically use a tiered memory hierarchy.

3.1 Tier 1: The Local (In-Process) Cache

• Technology: Python dict, C++ std::unordered_map.
• Latency: Nanoseconds (Reference access) to Microseconds.
• Capacity: Small (1GB - 10GB).
• Strategy: Hot-Keys Only. We use admission policies like TinyLFU, which admit a key only if its estimated access frequency beats the key it would evict, so only very popular items stay (e.g., the “Global Bias” features or a celebrity’s embedding).
• Gotcha: Cache Coherency. If Node A updates the cache, Node B doesn’t know. We accept this inconsistency.

3.2 Tier 2: The Distributed (Remote) Cache

• Technology: Redis (Cluster Mode), Memcached, Amazon ElastiCache.
• Latency: 1ms - 5ms (Network RTT).
• Capacity: Medium to Large (Terabytes).
• Strategy: Sharded & Replica. We shard data across 100+ nodes using Consistent Hashing.

3.3 Tier 3: The Persistent (SSD) Cache

• Technology: Aerospike, DynamoDB (DAX), NVMe Pools.
• Latency: 5ms - 20ms.
• Capacity: Petabytes.
• Role: Feature Store History. When you need to backfill data or fetch user history from 30 days ago.

3.4 Summary Table

Tier	Technology	Latency	Capacity	Use Case
L1	Python dict, C++ map	< 0.1ms	1-10GB	Hot keys, Global features
L2	Redis, Memcached	1-5ms	1-10TB	User features, Session data
L3	Aerospike, DAX	5-20ms	Petabytes	Historical features, Cold users

3.5 Eviction policies

Each tier needs a rule for what to drop when it is full:

Policy	Evicts	Good for
LRU (least recently used)	The key untouched for longest	Prediction caches, session data, anything with temporal locality
LFU (least frequently used)	The key with the fewest hits	Embeddings and features with skewed, stable popularity
TTL (time to live)	Keys older than a fixed age	Real-time features that go stale regardless of popularity

Production caches usually combine them: Redis offers allkeys-lru and allkeys-lfu eviction plus per-key TTLs, and in-process caches like Caffeine (Java) use W-TinyLFU. An in-process LRU takes a few lines in Python:

from collections import OrderedDict

class LRUCache:
    """O(1) get/put LRU cache backed by an ordered dict."""

    def __init__(self, capacity: int):
        self.capacity = capacity
        self.cache = OrderedDict()

    def get(self, key):
        if key not in self.cache:
            return None
        self.cache.move_to_end(key)  # mark as most recently used
        return self.cache[key]

    def put(self, key, value):
        self.cache[key] = value
        self.cache.move_to_end(key)
        if len(self.cache) > self.capacity:
            self.cache.popitem(last=False)  # evict least recently used

cache = LRUCache(capacity=2)
cache.put("a", 1); cache.put("b", 2)
cache.get("a")         # "a" is now most recent
cache.put("c", 3)      # evicts "b"
print(list(cache.cache))  # ['a', 'c']

4. Deep Dive: Consistent Hashing for Distributed Caching

When you scale Redis to 100 nodes, efficient key distribution is critical. Naive hashing hash(key) % N is broken. If one node dies, N changes to N-1, and 100% of keys are remapped. This causes a “Cache Miss Storm” that can take down your database.

4.1 The Solution: Virtual Nodes on a Ring

We map both Nodes and Keys onto the same integer hash ring (the code below uses 128-bit MD5 values).

• A Key belongs to the first Node found moving clockwise on the ring.
• Virtual Nodes: To balance load, each physical server is hashed to many points on the ring (100 to a few hundred is common; the code uses 100).
• Result: If a node dies, only its share of keys (about 1/N) moves to the neighbors. The other (N-1)/N stay put.

import hashlib
import bisect

class ConsistentHash:
    def __init__(self, nodes=None, replicas=100):
        self.replicas = replicas
        self.ring = {}
        self.sorted_keys = []
        if nodes:
            for node in nodes:
                self.add_node(node)

    def add_node(self, node):
        for i in range(self.replicas):
            key = self._hash(f"{node}:{i}")
            self.ring[key] = node
            bisect.insort(self.sorted_keys, key)

    def remove_node(self, node):
        for i in range(self.replicas):
            key = self._hash(f"{node}:{i}")
            del self.ring[key]
            self.sorted_keys.remove(key)

    def get_node(self, key):
        if not self.ring: return None
        h = self._hash(key)
        # Binary Search for the next node on the ring
        idx = bisect.bisect(self.sorted_keys, h)
        if idx == len(self.sorted_keys):
            idx = 0
        return self.ring[self.sorted_keys[idx]]

    def _hash(self, key):
        return int(hashlib.md5(key.encode()).hexdigest(), 16)

ring = ConsistentHash(["cache-a", "cache-b", "cache-c"])
keys = [f"user:{i}" for i in range(10_000)]
before = {k: ring.get_node(k) for k in keys}
ring.remove_node("cache-b")
moved = sum(before[k] != ring.get_node(k) for k in keys)
print(f"Keys moved after losing 1 of 3 nodes: {moved / len(keys):.0%}")  # about 33%

4.2 Replication for Read Scalability

Each shard can have read replicas.

• Primary: Handles writes.
• Replicas (2-3): Handle reads.
• Trade-off: Replication Lag. A write to Primary might not be visible on Replica for 10ms. For ML features, this is usually acceptable.

---

5. Embedding Caching: The ML-Specific Challenge

Standard caching assumes values are small (< 1KB). ML embeddings are large (512-dim float32 = 2KB) and there are billions of them.

5.1 Storage Optimization

• Quantization: Store embeddings as int8 instead of float32. Reduces storage by 4x, usually with a small recall loss; measure it on your own retrieval task.
• Dimension Reduction: Use PCA or learned projections to reduce 512-dim to 128-dim. 4x storage reduction.

5.2 Batch Fetching

When ranking 1000 items, we need 1000 embeddings.

• Naive: 1000 sequential GET calls, one network round trip each (about 1 s at 1 ms per round trip).
• Batched: one MGET or pipeline, one round trip (single-digit milliseconds). ```python

  r = redis.Redis(…)

  Bad: one network round trip per item

  embeddings = [r.get(f”item:{item_id}”) for item_id in item_ids]

Good: one round trip for the whole batch

embeddings = r.mget([f”item:{item_id}” for item_id in item_ids])

A pipeline does the same and also works for mixed commands

pipe = r.pipeline() for item_id in item_ids: pipe.get(f”item:{item_id}”) embeddings = pipe.execute()

### 5.3 Precomputation vs. On-Demand
-   **User Embeddings**: Change slowly (daily). Precompute and cache with 24h TTL.
-   **Item Embeddings**: Change on edit. Cache on first access, invalidate on update.
-   **Query Embeddings**: Ephemeral. Compute on-the-fly, cache for 5 minutes for repeat queries.

---

## 6. The "Thundering Herd" Problem & Solutions

Scenario: A celebrity tweets. Millions of users request their profile.
The cache key `user_profile:123` expires at 12:00:00.
At 12:00:01, 10,000 concurrent requests hit the cache -> Miss -> Hit the Database.
**The Database Dies.**

### 6.1 Solution 1: Request Coalescing (SingleFlight)
Only ONE request per key is allowed to compute. All other requests for that key **wait** (block) until the first one returns.
-   Implemented in Go (`singleflight`), Nginx (`proxy_cache_lock`).

### 6.2 Solution 2: Probabilistic Early Expiration (PER)
Each reader may refresh the value *before* it expires, with a probability that rises as expiry approaches. The XFetch rule from Vattani et al. ("Optimal Probabilistic Cache Stampede Prevention", VLDB 2015) scales the early window by how long the value takes to recompute:

```python
import math
import random
import time

def should_refresh_early(expiry_ts: float, recompute_secs: float, beta: float = 1.0) -> bool:
    """XFetch: refresh if now - recompute_secs * beta * ln(U) >= expiry, U ~ Uniform(0, 1]."""
    u = 1.0 - random.random()  # in (0, 1], avoids log(0)
    return time.time() - recompute_secs * beta * math.log(u) >= expiry_ts

# A value that takes 2 s to recompute and expires in 60 s is almost never
# refreshed early; one that expires in 1 s usually is.
now = time.time()
print(sum(should_refresh_early(now + 60, 2.0) for _ in range(1000)))  # close to 0
print(sum(should_refresh_early(now + 1, 2.0) for _ in range(1000)))   # several hundred

Because each reader rolls independently, usually only one or a few of them recompute, and the refresh happens before the hard expiry.

---

7. Semantic Caching for LLMs

LLM queries are expensive and slow.

• Q1: “Who is the President of France?”
• Q2: “Who is the current leader of France?” Traditional cache says MISS (Strings don’t match). Semantic Cache says HIT.

7.1 The Logic

1. Embed Query: v1 = BERT("Who is the President of France?")
2. Vector Search: Search Milvus/Faiss for nearest neighbor v2.
3. Distance Threshold: If Cosine_Similarity(v1, v2) > 0.95, return the cached answer associated with v2.

7.2 Architecture

• Store: Vectors in Faiss, Metadata in Redis.
• Tradeoff: 5ms Latency (Embedding + Search) vs 2000ms Latency (LLM Generation).
• Cost Savings: Depends on how repetitive the traffic is. FAQ-style support bots repeat a lot; open-ended chat repeats much less. Replay logged queries against a candidate threshold to estimate the hit rate before building it.
• Risks: A false hit returns a confidently wrong answer, so tune the threshold on labeled pairs. Scope cache keys by tenant and by anything that changes the answer (user permissions, retrieved documents, system prompt version), or one user’s answer can leak to another.

---

8. Feature Caching Strategies

In Recommender Systems, we need to fetch user features ([age, gender, clicks_7d]) and item features ([category, price, ctr]).

8.1 Row-Oriented vs Column-Oriented Caching

• Row Cache (Redis Strings): key=user:123 -> val=protobuf_blob.
  • Good for fetching single entities.
  • Serialization Formats: Protobuf (fast, compact), JSON (readable, slower), MessagePack (good balance).
• Column Cache (Redis Lists/Bitmaps): key=clicked_item:55 -> val=[user1, user2, user3].
  • Good for “Find all users who clicked X.”
  • Use Case: Co-occurrence matrices for collaborative filtering.

8.2 Feature Versioning

ML features evolve. You might change ctr_7d calculation from “clicks/views” to “clicks/(views + 100)”.

• Problem: Old cached values are now stale/incorrect.
• Solution: Include version in key: user:123:v2.
• Deployment: On deploy, warm cache with new version keys. Old keys expire naturally.

8.3 TTL Strategies by Feature Type

Feature Type	TTL	Rationale
Static (age, gender)	24h	Changes rarely
Slowly Changing (preferences)	1h	Updates daily
Real-Time (clicks_5min)	60s	Must be fresh
Computed (ML score)	5min	Expensive to compute

8.4 Write patterns: write-through, cache-aside, write-back and stream push

• Write-Through: The application writes to DB and Cache simultaneously. Safe but slow.
• Cache-Aside (Lazy Loading): Read DB on miss. Preferred for most features.
• Write-Back: Write to the cache and flush to the DB asynchronously. Fastest writes, but data is lost if the cache node fails before the flush, so avoid it for anything that is the system of record.
• Push-Based (Stream Hydration):
  • User clicks item.
  • Kafka Event triggers Flink Job.
  • Flink computes clicks_7d.
  • Flink Pushes update directly to Redis.
  • The Inference service serves strictly from Redis (never computes).

---

9. Failure Modes

9.1 Hot Keys

One key (e.g., “Justin Bieber”) gets more traffic than a single Redis shard can handle.

• Fix: Local Cache Replica. Detect hot keys and replicate them to L1 (Application Memory).
• Fix: Key Splitting. Store bieber as bieber:1, bieber:2. Randomly read one.

9.2 Network Bandwidth Saturation

Fetching 1000 items * 2KB embedding = 2MB payload per request. At 1000 QPS, that is 2 GB/s. This saturates a 10Gbps NIC.

• Fix: Compression (Zstd, LZ4).
• Fix: Quantization (Store float32 embeddings as int8).

---

10. Real-World Look: Facebook TAO

Facebook’s TAO (“The Associations and Objects”) is a graph-aware cache in front of MySQL. The TAO paper (USENIX ATC 2013) reports about a billion reads and millions of writes per second.

• Graph-Aware: It caches objects and associations (User -> Likes -> Page), not just opaque keys.
• Eventual Consistency: Follower caches can serve slightly stale data; writes go through a leader tier.
• Leases: Facebook’s earlier memcache work (Nishtala et al., NSDI 2013) hands a lease token to the first client that misses on a key, so only that client refills it while others wait or get slightly stale data.

---

11. Cache Invalidation: The Two Hardest Problems

Phil Karlton famously said: “There are only two hard things in Computer Science: cache invalidation and naming things.”

11.1 Time-Based Invalidation (TTL)

The simplest strategy. Set EXPIRE key 3600 (1 hour).

• Pros: Simple. No coordination needed.
• Cons: User might see stale data for up to 1 hour. Hot features like “Live Stock Price” cannot tolerate this.

11.2 Event-Based Invalidation (CDC)

Use Change Data Capture (Debezium, Maxwell) to listen to database changes.

• User updates their profile in PostgreSQL.
• Debezium captures the WAL event.
• Kafka delivers {user_id: 123, type: UPDATE} to our Invalidation Consumer.
• Consumer calls DEL user:123 on Redis.
• Next read fetches fresh data from DB and populates cache.

Implementation Pattern:

import json

from kafka import KafkaConsumer
import redis

consumer = KafkaConsumer('db.users', bootstrap_servers=['kafka:9092'])
r = redis.Redis()

for message in consumer:
    event = json.loads(message.value)
    user_id = event['after']['id']

    # Invalidate the cache for this user
    r.delete(f"user:{user_id}")
    r.delete(f"user_embedding:{user_id}")

    print(f"Invalidated cache for user {user_id}")

11.3 The Dual-Write Anti-Pattern

Never do this in production:

# WRONG: Race Condition!
db.update(user)
cache.delete(user_key)

If cache.delete fails (network hiccup), the cache stays stale until the key’s TTL expires, or forever if it has none.

Safer Pattern:

# RIGHT: Transactional Outbox
db.update(user)
db.insert_into_outbox(event)

# Separate worker polls outbox
for event in db.poll_outbox():
    cache.delete(event.key)
    db.delete_from_outbox(event.id)

---

12. Monitoring and Observability

You cannot manage what you cannot measure. A production cache requires rigorous observability.

12.1 Key Metrics to Track

Metric	Target	Alarm Threshold
Cache Hit Ratio (CHR)	> 90%	< 80%
p99 Latency (Redis)	< 5ms	> 15ms
Memory Utilization	< 80%	> 90%
Eviction Rate	Low	Sudden Spike
Connection Pool Exhaustion	0	> 0

12.2 Instrumenting Your Cache Client

Wrap your Redis calls with metrics.

import time
from prometheus_client import Histogram, Counter

CACHE_LATENCY = Histogram('cache_latency_seconds', 'Cache operation latency', ['op', 'status'])
CACHE_HITS = Counter('cache_hits_total', 'Number of cache hits')
CACHE_MISSES = Counter('cache_misses_total', 'Number of cache misses')

class InstrumentedCache:
    def __init__(self, redis_client):
        self.redis = redis_client

    def get(self, key):
        start = time.time()
        value = self.redis.get(key)
        duration = time.time() - start

        if value is None:
            CACHE_MISSES.inc()
            CACHE_LATENCY.labels(op='get', status='miss').observe(duration)
        else:
            CACHE_HITS.inc()
            CACHE_LATENCY.labels(op='get', status='hit').observe(duration)

        return value

12.3 Dashboards You Need

1. Hit Ratio over Time: Detects model drift or changed usage patterns.
2. Latency Heatmap: Identifies slow shards or network issues.
3. Memory Fragmentation: Redis can use 2x memory due to fragmentation. Track mem_fragmentation_ratio.

---

13. KV Cache for Transformers (LLM Inference)

This is a specialized cache that lives inside the model inference, not in the infrastructure.

13.1 The Problem

In decoder-only Transformers (Llama 3, for example), generating 100 tokens requires computing Attention N times. Each Attention computation re-uses the same Key and Value vectors for the previous tokens. Without caching, you recompute these KV pairs at every step. This is O(N^2) for N tokens.

13.2 The KV Cache Solution

We store the Key and Value matrices in GPU VRAM.

• Token 1: Compute K1, V1. Store.
• Token 2: Load K1, V1. Compute K2, V2. Store K1, K2, V1, V2.
• Token 50: Load K1..K49, V1..V49. Compute K50, V50. Store.

VRAM Cost: KV cache bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per value. For a Llama 70B-class model (80 layers, 8 KV heads of dimension 128 with grouped-query attention, FP16) that is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, about 320 KB per token. One 4,096-token sequence takes about 1.3 GB, and a batch of 32 such sequences about 43 GB. At high concurrency the KV cache, not the weights, sets the limit. The KV cache for MoE post works through the same arithmetic for sparse models.

13.3 Optimization: Paged Attention (vLLM)

vLLM (from UC Berkeley) treats KV Cache like OS Virtual Memory.

• Pages: Divide the cache into fixed-size blocks (vLLM defaults to 16 tokens per block).
• On-Demand Allocation: Only allocate pages when needed.
• Sharing: If two prompts share a prefix, they can share the KV pages for that prefix.
• Result: In their 2023 launch benchmarks, the vLLM team reported up to 24x higher throughput than Hugging Face Transformers.

---

14. Worked example: feature caching for a large recommender

This is an illustrative design, not a description of any specific company’s system. Assume hundreds of millions of users and a recommender that fetches about 1,000 features per request.

14.1 The Architecture

• L1 (Application): 5GB LRU cache per pod. Holds global features (e.g., “is_weekend”).
• L2 (Redis Cluster): 500 nodes, 10TB capacity. Holds user features.
• L3 (RocksDB-backed SSTable store): Hourly snapshots of the feature store for cold users.

14.2 Key Optimizations

1. Feature Packing: Instead of 1000 GET calls, pack all features for a user into a single Protobuf blob. 1 GET per user.
2. Speculative Warming: When a user logs in, a background job pre-fetches their features into L2 before the first recommendation request.
3. Tiered Eviction: Global features have TTL=infinity. User features have TTL=7 days. Ephemeral features (session data) have TTL=10 minutes.

14.3 Targets to design for

• p50 Latency: about 2ms (L1 hit).
• p99 Latency: under 10ms (L2 hit).
• Cache Hit Ratio: high 90s percent, so the SSD tier sees only a small fraction of reads.

---

15. Advanced Topic: Negative Caching

What happens when you query for a user that doesn’t exist?

• Request: GET user:99999999
• Redis: MISS
• Database: SELECT * FROM users WHERE id=99999999 -> Empty.
• Problem: Next request repeats this expensive DB query.

15.1 The Solution: Cache the Null

Store a special sentinel value:

value = db.get_user(user_id)
if value is None:
    cache.set(f"user:{user_id}", "NULL", ttl=300)  # Cache the non-existence
else:
    cache.set(f"user:{user_id}", serialize(value), ttl=3600)

On read:

def get_user(user_id):
    cached = cache.get(f"user:{user_id}")
    if cached == "NULL":
        return None  # Known not to exist: don't hit DB
    if cached is not None:
        return deserialize(cached)
    return load_and_cache(user_id)  # the write path shown above

15.2 Danger: Botnet Attacks

If an attacker queries millions of random non-existent IDs, you fill your cache with “NULL” entries, evicting real data.

• Fix: Keep a Bloom Filter of existing IDs in front of the cache. If it says “definitely not present”, return immediately; only check the cache and DB when it says “maybe exists.”

---

16. Cache Warming Strategies

A cold cache is a dangerous cache. The first 5 minutes after deployment can see massive latency spikes.

16.1 Pre-Warming on Deploy

Before routing traffic to a new pod:

1. Query Log Replay: Replay the last hour of cache access logs against the new pod.
2. Feature Snapshot Load: Load a snapshot of the top 10,000 most popular features directly into L1.
3. Health Check with Warm Threshold: Don’t mark the pod “Ready” until Cache Hit Ratio > 50%.

16.2 Shadow Traffic

Before cutting over to a new cache cluster:

• Fork Traffic: Send a copy of all production requests to the new cluster (without returning results to users).
• Goal: Pre-populate the cache with real access patterns.
• Duration: Run for 10 minutes before cutting over.

16.3 Scheduled Warming for Time-Bound Features

Features like day_of_week or is_holiday are known in advance.

• CRON Job: At midnight, compute and push all time-based features for the next 24 hours.
• Result: Zero cold-start latency for the first recommendation of the day.

---

17. Production Best Practices

Practices that hold up in production:

17.1 Separate Caches by SLA

Don’t mix real-time serving cache with batch processing cache.

• Serving Cache: Low TTL, High Memory Priority, Strict SLA.
• Training Cache: High TTL, Lower Priority, Best Effort.
• Risk: A batch job could evict serving data.

17.2 Use Client-Side Connection Pooling

Opening a new TCP connection per call adds a handshake round trip (more with TLS).

• Reuse one client per process; redis-py pools connections through ConnectionPool by default.
• Set max_connections based on your QPS and concurrency.

17.3 Monitor Memory Fragmentation

Redis uses jemalloc. Heavy SET/DEL cycles lead to fragmentation.

• Metric: mem_fragmentation_ratio > 1.5 is a warning sign.
• Fix: Use MEMORY DOCTOR to diagnose and enable activedefrag yes (supported with the default jemalloc allocator).

17.4 Implement Circuit Breakers

If Redis is down, don’t hammer it with retries.

• Use a Circuit Breaker (resilience4j, pybreaker).
• When open, serve from L1 cache only or return a default/fallback prediction.

---

18. Key Takeaways

1. Caching is the First Line of Defense: Protect your expensive GPU inference layers with a multi-tier hierarchy.
2. Consistency vs. Latency: In ML, it is often better to serve a stale prediction (eventual consistency) than to wait.
3. Context Matters: Choose your eviction policy based on the “intent” of the data. LFU for popular items, LRU for temporal access.
4. Semantic Search is the Future: Move beyond exact-match keys to vector similarity for LLM caching.
5. Invalidation is Hard: Use CDC/Event-based invalidation, not dual-writes.
6. Monitor Everything: Cache Hit Ratio, Latency p99, Eviction Rate are your core metrics.

FAQ

What is semantic caching for LLMs and how does it work?

Semantic caching uses vector embeddings to find queries that are semantically similar rather than string-identical. When a new query’s embedding has a cosine similarity above a threshold (typically 0.95) with a cached query, the cached answer is returned instead of calling the LLM. This saves expensive generation time and typically cuts 30-40% of compute costs in production chat applications.

How do you solve the thundering herd problem in ML caching?

Two proven approaches: Request Coalescing (SingleFlight) allows only one request per cache key to compute while others wait for the result. Probabilistic Early Expiration refreshes cache entries before they expire based on a probability function, spreading the refresh load over time so thousands of concurrent requests never simultaneously hit the database.

What is consistent hashing and why is it important for distributed caches?

Consistent hashing maps both nodes and keys to a ring structure. A key belongs to the first node found clockwise on the ring. When a node fails, only 1/N of keys need remapping instead of 100% with naive modulo hashing. Virtual nodes (1,000 per physical server) ensure even load distribution and prevent data hotspots across the cluster.

How does PagedAttention optimize KV cache memory for LLM inference?

PagedAttention treats KV cache like OS virtual memory, dividing it into fixed-size pages (typically 256 tokens) allocated on demand. This eliminates memory fragmentation from pre-allocating maximum sequence lengths. Shared prefixes between requests can point to the same physical pages, achieving up to 24x higher throughput compared to naive HuggingFace implementations.