The Workload-Router-Pool: how vLLM thinks about fleet inference
TL;DR: The March 2026 vision paper “The Workload-Router-Pool Architecture for LLM Inference Optimization” (arXiv:2603.21354, from the vLLM Semantic Router project) frames fleet LLM inference as three interdependent dimensions: Workload (what you serve), Router (how requests are dispatched), and Pool (where inference runs). Agentic tasks make 3 to 10x more LLM calls per user request than chat (the paper cites ServeGen for this), break KV-cache locality under round-robin routing, and need session awareness that a stateless load balancer does not have. The WRP framework shows why routing policy, workload mix, and pool topology must co-evolve; optimizing one constrains the others.

Most teams start with a single vLLM instance. Traffic grows. They add a second instance and route with round-robin. Traffic grows more. They add a load balancer, then a model router, then a KV-cache pool, each addition patching a new symptom. The system works. Barely, expensively, with a growing list of exceptions.
The March 2026 paper “The Workload-Router-Pool Architecture for LLM Inference Optimization” (arXiv:2603.21354, vLLM Semantic Router project) names the three dimensions that must co-evolve to get from patched single-server to coherent fleet.
The three dimensions
Workload characterizes what the fleet actually serves. Not all LLM requests are alike:
- Chat vs. agentic: a single-turn question takes one LLM call; an agentic loop making 10-30 tool calls takes roughly that many
- Prefill-heavy vs. decode-heavy: a request with a 32K-token context but a short answer stresses the GPU differently than a short prompt generating a long response
- Warm vs. cold start: a request that shares a prefix with recent requests can reuse KV-cache; one with a novel prefix cannot
- Single-turn vs. multi-turn: stateless requests can go anywhere; multi-turn sessions benefit from instance affinity
Router determines how requests are dispatched across the pool. The paper identifies four routing strategies:
| Strategy | Mechanism | When to use |
|---|---|---|
| Static semantic rules | Domain/keyword classification | Stable workload mix, few models |
| Online bandit | Multi-armed bandit learning from latency feedback | Dynamic workload, cost/latency trade-off |
| RL-based model selection | Learned policy over model quality and cost | Complex multi-objective routing |
| Quality-aware cascading | Route to small model first, escalate if confidence low | Cost reduction with quality floor |
Pool defines where inference runs:
- Homogeneous vs. heterogeneous GPU clusters (mixing H100s, A100s, consumer GPUs)
- Disaggregated prefill/decode: separate servers for the two inference phases
- KV-cache topology: per-instance, shared pool, or hierarchical offload
The key architectural insight: these three dimensions are not independent. Your pool topology constrains which routing strategies are viable. Your routing strategy depends on your workload mix. You cannot design any one dimension in isolation. The paper maps the interactions in a 3x3 matrix and proposes 21 research directions where the cells are still open.
Why agentic workloads are the stress test
Single-turn chat requests are stateless and roughly uniform in cost. An agentic loop is neither.
A user asking a question generates one LLM call. An agent solving a coding task generates a planning call, several tool-use calls, a synthesis call, and possibly a self-correction loop. The WRP paper, citing the ServeGen workload study, puts agents at 3 to 10x more LLM calls per user request than chatbots. At scale, this is a fleet sizing problem disguised as an application feature.
The KV-cache problem is subtler. In a single-turn deployment, every request starts fresh with little KV-cache reuse expected beyond a shared system prompt. In an agentic loop, each iteration resends the same growing context. If iteration 1 lands on server A and iteration 2 lands on server B (because round-robin doesn’t know they’re related), server B has no context to reuse. It processes the full prompt from scratch, burning compute and latency that a warm instance would have skipped.
sequenceDiagram
participant Client
participant Router
participant Server A
participant Server B
Note over Router: Round-robin (naive)
Client->>Router: Agentic iteration 1
Router->>Server A: Dispatch (builds KV-cache)
Client->>Router: Agentic iteration 2 (same session)
Router->>Server B: Dispatch (KV-cache miss!)
Server B-->>Client: Recomputes full context
Note over Router: Cache-aware routing
Client->>Router: Agentic iteration 1
Router->>Server A: Dispatch (builds KV-cache)
Client->>Router: Agentic iteration 2 (same session)
Router->>Server A: Dispatch (KV-cache hit on shared prefix)
Server A-->>Client: Prefill only the new tokens
Keeping a session on a warm instance is the job of a cache-aware scheduler such as llm-d’s prefix-cache-aware routing. The WRP paper lists session-affinity routing as a proposal for keeping KV-cache locality, not as something the Semantic Router does today. What the vLLM Semantic Router (v0.1 Iris, January 2026; v0.4 shipped in September 2026) does is the model-selection side of the Router dimension: it classifies requests from domain, keyword, embedding and other signals and picks the model and reasoning mode. Its reasoning-aware routing paper (arXiv:2510.08731) reported a 10.2 percentage point accuracy improvement on MMLU-Pro, 47.1% lower latency and 48.5% fewer tokens than direct inference with vLLM, by applying reasoning only to queries that benefit from it.
Teams that run several models make these routing decisions on every request, so a poor policy gets expensive quickly.
The disaggregation question
Disaggregated prefill-decode is the Pool dimension’s most significant architectural choice. The argument:
LLM inference has two distinct phases with incompatible resource profiles. Prefill (processing the input prompt in parallel) is compute-bound: essentially matrix multiplication that scales well with more compute. Decode (generating tokens one at a time, autoregressively) is memory-bandwidth bound, since each token generation reads the full KV-cache and the model weights.
Running both phases on the same server forces each to compete for the same resources. Separating them onto dedicated servers allows each to be optimized and scaled independently.
The gains are real but workload-dependent. DistServe (arXiv:2401.09670, OSDI 2024) reported serving up to 7.4x more requests, or meeting a 12.6x tighter latency SLO, compared with state-of-the-art serving systems, while keeping over 90% of requests within their latency targets. llm-d and NVIDIA Dynamo both support disaggregated serving for this reason.
The cost is KV-cache locality. When prefill and decode run on separate servers, the KV-cache generated during prefill must transfer to the decode server, and a prefill tier that ignores which prefixes are already cached wastes work. Together AI’s cache-aware disaggregated inference post reports up to about 40% higher sustainable throughput than conventional prefill/decode deployments on a synthetic coding-agent workload with heavy prefix reuse, by adding a cache-aware router and a separate tier for cold prefills.
The decision tree:
Prefill latency (TTFT) dominant, or prefill bursts hurting decode latency?
├─ Yes + long prompts?
│ └─ Consider disaggregation (DistServe: up to 7.4x more requests within SLO).
│ But: measure KV-cache hit rate first.
│ If prefix reuse is high, make the router cache-aware
│ so disaggregation does not throw the hits away.
└─ No: decode latency (TPOT) dominant?
└─ Scale decode replicas horizontally.
Keep prefill co-located for cache locality.
The fleet scaling decision
Single-server benchmarks explain why teams start with vLLM. In Red Hat’s comparison on an A100 with Llama 3.1 8B, vLLM peaked at 793 tokens/sec against Ollama’s 41, with P99 latency of 80 ms versus 673 ms. But those numbers come from a single server under controlled load.
Fleet scaling introduces new problems the single-server numbers don’t address:
Session affinity. Multi-turn and agentic requests need instance affinity. Stateless load balancing breaks them. You need either sticky sessions (which limit load balancing flexibility) or shared KV-cache (which adds infrastructure complexity).
Workload heterogeneity. A fleet serving both short chat requests and long agentic tasks has two different latency SLAs, two different cost profiles, and two different optimal batching strategies. A single homogeneous pool optimized for one breaks the other.
Model multiplicity. When you run multiple models in production, the Router dimension becomes an economic decision: small models for simple tasks, large models for complex ones, specialized models for domain-specific work. Quality-aware cascading (route to a small model first, escalate on low confidence) can cut cost when most requests are easy; measure the quality floor before you rely on it. The build vs buy framework covers when running this fleet yourself is worth it at all.
A rough progression, as a starting point rather than thresholds from the paper:
| Stage | Architecture | Typical tools |
|---|---|---|
| One model, modest traffic | Single vLLM deployment; tune batch size and prefix caching | vLLM |
| Several models or mixed workloads | Model selection plus prefix-aware routing, autoscaling | vLLM Semantic Router, llm-d, KEDA |
| Large fleet, long prompts | Disaggregated prefill/decode, shared or tiered KV cache, multi-zone | llm-d, NVIDIA Dynamo |
For the routing layer specifically, see vLLM’s semantic router, which separates model selection from prefix-aware cache routing. The WRP paper extends that to show how the Router design is constrained by the Workload profile and must co-evolve with Pool topology.
Key takeaways
- The WRP framework (arXiv:2603.21354) names three interdependent fleet inference dimensions: Workload, Router, Pool. Optimizing any one without the others produces a locally optimal, globally broken system.
- Agentic workloads make 3 to 10x more LLM calls per user request than chat (WRP, citing ServeGen) and need cache-aware, session-aware routing, not round-robin.
- Reasoning-aware model selection (arXiv:2510.08731) gave a 10.2 point MMLU-Pro gain and 47.1% lower latency than direct vLLM inference. KV-cache reuse comes from a different mechanism: prefix-aware routing.
- Disaggregated prefill-decode can raise goodput substantially (DistServe: up to 7.4x more requests within SLO) but moves KV-cache across machines. Pair it with cache-aware routing when prefix reuse is high.
- When you run several models, the Router dimension doubles as cost optimization: quality-aware cascading routes easy tasks to small models and escalates complex ones.
FAQ
What is the Workload-Router-Pool architecture? WRP (arXiv:2603.21354, vLLM Semantic Router project, March 2026) frames fleet LLM inference as three dimensions: Workload (what you serve), Router (how requests are dispatched), and Pool (where inference runs). The key insight is their interdependence: routing policy, workload mix, and pool topology must co-evolve.
Why do agentic workloads break naive round-robin routing? Agentic multi-step requests make many LLM calls per user request and are stateful: each iteration resends context from prior ones. Round-robin sends iterations to arbitrary instances, causing KV-cache misses. Cache-aware routing keeps sessions on a warm instance or shares KV-cache across the pool.
What is disaggregated prefill-decode and when does it help? Separating the compute-intensive prefill phase from the memory-bandwidth-bound decode phase onto dedicated servers lets each scale independently and stops long prefills from stalling decode. DistServe reported up to 7.4x more requests served within latency targets. The cost is moving KV-cache between machines, so pair it with cache-aware routing when prefix reuse is high.
What is the vLLM Semantic Router and how does it relate to WRP? The vLLM Semantic Router (v0.1 Iris, January 2026) is a concrete Router implementation focused on model selection: it classifies requests from domain, keyword, embedding and other signals and picks a model and reasoning mode, with plugins for semantic caching and safety filtering. WRP is the architectural framework; Semantic Router is one Router implementation within it.
When should a team move from single-server to fleet LLM inference? Move when one deployment no longer meets your latency targets under production load, when you serve several models and need to choose between them per request, when your workload mixes short chat and long agentic requests with different SLAs, or when models and KV cache no longer fit on one node. Start with prefix-aware routing and autoscaling; add disaggregation and shared KV cache when long prompts dominate.
Further reading
- The Workload-Router-Pool Architecture for LLM Inference Optimization: the vision paper this post is built on (vLLM Semantic Router project, March 2026)
- vLLM Semantic Router v0.1 Iris Release: the production Router implementation
- Cache-Aware Disaggregated Inference: Together AI’s analysis of KV-cache locality in disaggregated serving