13 minute read

Load balancing assumes requests are interchangeable. They’re not.

A request differs from its neighbours in two ways that matter to an LLM fleet. First, what it needs: a math question may deserve a reasoning model, a formatting task does not, and a jailbreak attempt should not reach a model at all. Second, what it shares: many requests start with the same system prompt, tool schema or document, and an instance that has already processed that prefix holds its KV cache. Round-robin ignores both.

vLLM’s Semantic Router addresses the first. It reads each request, classifies it, and picks the model (and reasoning mode) that should serve it. The second is the job of prefix-cache-aware routing, such as llm-d’s scheduler, which sends a request to the instance already holding its prefix. The two are easy to conflate, and the distinction decides which one your deployment needs.

TL;DR

Round-robin load balancing treats every LLM request as interchangeable. vLLM’s Semantic Router (v0.1 “Iris”, January 2026) classifies requests from signals such as domain, keywords and embedding similarity, and maps each one to a model, pool and configuration. Its headline result is about model selection: applying reasoning only to queries that need it gave a 10.2 point accuracy gain on MMLU-Pro with 47.1% lower latency and 48.5% fewer tokens than sending everything through reasoning (arXiv:2510.08731). KV cache reuse is a separate mechanism. It needs an exact token-prefix match, and the large cache wins come from prefix-aware routing: Red Hat measured an 87.4% cache hit rate and 88% faster time-to-first-token on warm hits with llm-d. For the LLM serving fundamentals this builds on, see LLM serving infrastructure.

An automated package sorting conveyor with similar-colored items clustering into the same few destination bins rather than distributing randomly, a...


Why load balancing isn’t enough for LLM inference

Standard load balancers optimize for utilization. They don’t know that two requests share a 2,000-token system prompt. The immediate problem: every request that hits a cold instance pays full prefill cost. For a 2,000-token shared prefix on a mid-size model, that’s compute you’re doing over and over, per request, per instance, with zero reuse.

The core issue is that LLM inference has a property no traditional service does: KV cache warmth. When a request arrives at an instance that already processed the same prefix, the attention keys and values for those tokens are already in GPU memory, and the engine skips prefill for them. Hosted APIs reflect this in price: cached input tokens are commonly billed at a large discount, up to 90% off. Round-robin routing throws this away by design.

The gap widens with context length. As prompt sizes grow (agent prompts often run past 16K tokens), the cost difference between a warm hit and a cold request grows with them. For illustration, assume $1 per million input tokens and a 90% discount on cached tokens: a 16K-token prompt costs about $0.016 cold and $0.0016 warm, so each miss wastes about $0.014. At 10,000 agent calls per hour, that is roughly $140 per hour lost to routing alone.

What makes this tractable is that production workloads are more correlated than they look. RAG pipelines hit the same document chunks repeatedly. Agent clusters share system prompts and tool schemas. Chatbots with persona configuration share hundreds of tokens before the first user message. The signal is there. Standard load balancers just don’t read it.

ROUND-ROBIN ROUTING (cache blind), 3 instances

Request A (system_prompt + query_1) ──→ Instance 1  [cold, full prefill]
Request B (system_prompt + query_2) ──→ Instance 2  [cold, full prefill]
Request C (system_prompt + query_3) ──→ Instance 3  [cold, full prefill]
Request D (system_prompt + query_4) ──→ Instance 1  [warm, by luck]

Result: 1 of 4 hits on the shared prefix; the odds fall as instances grow

PREFIX-AWARE ROUTING (cache-aware)

Request A (system_prompt + query_1) ──→ Instance 1  [cold, full prefill]
Request B (system_prompt + query_2) ──→ Instance 1  [warm hit on system_prompt]
Request C (system_prompt + query_3) ──→ Instance 1  [warm hit on system_prompt]
Request D (system_prompt + query_4) ──→ Instance 1  [warm hit on system_prompt]

Result: 3 of 4 hits on the shared prefix

Note what makes the second case work: the requests share an identical token prefix. A KV cache entry is only reusable when every token before it matches. Two requests that mean the same thing in different words share no KV cache at all.


What the Workload-Router-Pool architecture does

The WRP framework, introduced in arXiv:2603.21354 by Huamin Chen et al. (March 2026), gives a name and structure to the full problem space. It splits inference optimization into three interacting dimensions:

Workload describes what the fleet actually serves. Not all requests are the same. Chat is single-turn, short, and decode-heavy. Agent workloads make 3-10x more LLM calls per user request than chatbots (the WRP paper cites this from ServeGen) and are often prefill-heavy. A workload characterization layer that classifies incoming requests by type, length profile, and session structure is the prerequisite for everything else.

Router determines how each request gets dispatched. This is where semantic routing lives. The router extracts signals and maps each request to a (model, pool, configuration) tuple. In the Iris v0.1 release (January 2026), the signal layer supports six signal types: domain, keyword, embedding, factual, feedback, and preference. These compose through AND/OR logic with priorities rather than a fixed decision tree, making the routing policy configurable without code changes. The router also carries plugins for semantic caching, jailbreak and PII detection, and hallucination detection. The project has moved quickly since Iris (v0.4 “Hermes” shipped in September 2026), so check current docs for the signal list.

Pool defines where inference runs. GPU topology, disaggregated prefill/decode boundaries, and KV cache layout all belong here. The WRP paper maps the interactions between the three dimensions in a 3x3 matrix and proposes 21 open research directions at the intersections. One of them is exactly the gap this post is about: the paper suggests session-affinity routing to keep a session’s KV cache local, as a proposal rather than a shipped feature.

graph LR
    Client["Client Request"] --> SR["Semantic Router\n(Signal Extraction)"]
    SR --> |"Domain, keyword,\nembedding signals"| RD["Routing Decision\n(model, pool, config)"]
    RD --> |"Hard reasoning query"| P1["Pool A\nReasoning model"]
    RD --> |"Simple query"| P2["Pool B\nFast model"]
    RD --> |"Similar question seen"| SC["Semantic cache\n(cached response)"]
    P1 --> PA["Prefix-aware scheduler\npicks warm instance"]
    P2 --> PA
    PA --> OUT["Response"]
    SC --> OUT

The classifiers behind the signals are small encoder models. The WRP paper describes them as mmBERT LoRA adapters chosen to keep routing low-latency and low-resource, and the Iris release notes describe LoRA-based classifiers that share one base model across tasks. Classification adds a small fixed cost to each request, which you should measure against your TTFT budget.


Where the gains actually show up

The gains aren’t uniform, and they come from two different mechanisms.

Model selection for reasoning workloads. This is the Semantic Router’s core result. The arXiv:2510.08731 paper (NeurIPS 2025 ML for Systems workshop) classifies queries by whether they benefit from reasoning and applies reasoning only where it helps. Against direct inference with vLLM, it reported a 10.2 percentage point accuracy improvement on MMLU-Pro, 47.1% lower latency and 48.5% fewer tokens.

Prefix cache reuse for multi-agent systems. This comes from prefix-aware routing, not from semantic similarity. Red Hat’s benchmark of KV cache-aware routing with llm-d (October 2025) reported an 87.4% cache hit rate across 4,776 queries with shared prefixes. TTFT for warm cache hits was 340 ms against 2,850 ms cold, an 88% reduction from routing policy alone. The llm-d scheduler scores each instance by the longest run of matching KV blocks from the start of the prompt.

Multi-agent workloads benefit because agent calls within a session share structural context. Every call in a ReAct loop shares the same system prompt, the same tool schema, often the same document context. These aren’t coincidentally similar; they’re architecturally correlated by design. Routing blindly breaks the correlation. Routing by prefix preserves it.

RAG pipelines have a similar property when the retrieved chunks appear in the same order at the start of the prompt. When ten users ask questions about the same document, a prefix-aware router can direct all ten to the instance where the chunk’s KV entries are already hot. Separately, the WRP paper reports a token-budget pool routing scheme reducing GPU instances by 17-39% on production traces.

Chatbots with long system prompts capture a simpler version of the same benefit, since every request shares the persona configuration and behavior instructions. The llm-d team reports precise prefix-cache-aware scheduling delivering 57x faster response times than approximate scheduling on its benchmark (llm-d blog).


When to add a semantic router vs. simpler alternatives

Not every deployment needs this. A single-model endpoint serving diverse, independent queries gets nothing from model selection, because there is only one model to select. Adding classification overhead without a choice to make is pure cost.

The decision framework is roughly this:

Workload Semantic router (model selection) Prefix-aware routing (KV reuse)
Several models or reasoning modes, mixed difficulty Yes If prompts share prefixes
Multi-agent loops on one model Only for guardrails Yes (shared system prompt and tools)
RAG with shared corpus Optional Yes, if chunks lead the prompt
Single-model chatbot (long system prompt) Optional (guardrails, semantic cache) Yes
Single-turn classification / extraction Rarely Rarely

For KV reuse, prefix-aware routing (matching exact token prefixes or block hashes) is the right tool, and vLLM’s production stack and llm-d support it. Approximate similarity has a different use: the Semantic Router’s semantic cache can return a stored answer for a paraphrased question without calling a model at all. That saves the whole request, but it is a response cache with its own correctness risks, not KV reuse.

There is no universal breakeven. Compare the classifier’s added latency and GPU cost with what better model choices save, measured on your own traffic.

The WRP paper’s FleetOpt result adds a pool-side angle: co-designing the fleet up front cost 3.1-6.4% less than retrofitting compression onto an existing fleet. For large deployments, the pool design decision matters as much as the routing policy. They’re not independent choices; the Workload-Router-Pool post covers that side. If you are still deciding whether to run your own models at all, start with build vs buy for AI.

For the broader picture of how multi-agent systems create these correlated workloads in the first place, see Scaling multi-agent systems. For observability tooling that helps you measure cache hit rates in production, see Observability and tracing for AI agents.


FAQ

What is vLLM’s Semantic Router? vLLM’s Semantic Router is a request router that sits between users and model instances, classifies each request from signals such as domain, keywords and embedding similarity, and picks the model, pool and configuration to serve it. It also offers semantic caching and jailbreak and PII filtering. Its first major release, v0.1 “Iris”, shipped in January 2026.

Does semantic routing improve KV cache hit rates? Not directly. KV cache reuse needs an identical token prefix, so the hit-rate gains come from prefix-aware routing, which sends a request to the instance that already holds its prefix. Red Hat’s llm-d benchmark reported an 87.4% cache hit rate and 88% faster TTFT on warm hits with that approach. Semantic routing helps indirectly when it sends requests that share a system prompt to the same pool.

What is the Workload-Router-Pool (WRP) architecture? WRP is the three-part framework from arXiv:2603.21354 (March 2026) for organizing LLM inference optimization. Workload characterizes request types: chat vs. agent, single-turn vs. multi-turn, prefill-heavy vs. decode-heavy. Router covers dispatch strategy: semantic rules, bandit adaptation, RL-based model selection, cascading. Pool covers where inference runs: GPU topology, disaggregated prefill/decode, KV cache layout.

When does semantic routing make sense vs. simpler alternatives? Semantic routing pays off when you have a real choice of models or reasoning modes and request difficulty varies, or when you want guardrails and a semantic response cache in front of your models. If you serve one model and want KV reuse, prefix-aware routing is simpler and does the job.

How should multi-agent workloads be routed? Multi-agent systems make 3-10x more LLM calls per user request than chatbots, and many of those calls share an identical prefix: the same system prompt, tool schema or document chunk. Route them with a prefix-aware scheduler so repeated prefixes land on a warm instance, and add a semantic router in front only if different agent steps should go to different models.


Sources: arXiv:2603.21354, “The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project”, Huamin Chen et al., March 2026. arXiv:2510.08731, “When to Reason: Semantic Router for vLLM”, NeurIPS 2025 ML for Systems workshop. vLLM Semantic Router v0.1 Iris release, January 2026.