Streaming Real-Time Agents
“Waiting 10 seconds for a thoughtful answer is okay. Waiting 10 seconds for a blank screen is broken.”
TL;DR
Streaming transforms the agent UX from “wait and hope” to progressive disclosure. Server-Sent Events (SSE) are the standard for LLM streaming: simpler than WebSockets and friendlier to firewalls and load balancers. The core engineering challenge is leakage control: building a state machine that streams final answer tokens while hiding raw tool calls and intermediate reasoning. In production, edge buffering from Nginx or Cloudflare silently kills streaming unless explicitly disabled. Visual status updates (“Searching Google…”) reduce perceived latency even when actual latency is high. For a deeper look at real-time pipeline architecture including backpressure and the Actor Model, see Real-Time Agent Pipelines.

1. Introduction
Human conversation is streamed. We start processing the first word of a sentence before the speaker finishes the paragraph. Early LLM applications waited for the full generation (Stop Token) to complete before sending a JSON response.
- Old Way: Request -> Wait 15s -> Show 500 words. (User thinks app crashed).
- New Way: Request -> Wait 0.5s -> Show “The”… “quick”… “brown”…
For AI Agents, streaming is harder because they have “Thought Steps” (internal monologues) that the user shouldn’t see, interspersed with “Final Answers”.
2. Core Concepts: Protocols
How do we push data to the browser?
- Short Polling: Client asks “Done?” every 1s. (Inefficient).
- WebSockets: Bi-directional, full-duplex TCP. Good for real-time gaming, overkill for chat. Hard to load balance (stateful connections).
- Server-Sent Events (SSE): The standard for LLMs.
- One-way HTTP connection.
- Server keeps socket open and pushes
data: ...chunks. - Simple to implement with standard Load Balancers.
3. Architecture Patterns: The Stream Transformer
We need an architecture that transforms raw LLM tokens into structured Agent Events.
[LLM (OpenAI)]
| (Stream of Tokens)
v
[Agent Parser Logic]
| Detects: "Action: Search" -> WAIT
| Detects: "Observation: 42" -> WAIT
| Detects: "Final Answer: The..." -> STREAM
v
[Frontend (React)]
The key challenge: Leakage. We don’t want to stream the raw JSON braces of a tool call to the user. We only want to stream the “final_answer”.
4. Implementation Approaches
4.1 The Generator Pattern (Python)
Python yield is perfect for this.
import json
from openai import AsyncOpenAI
client = AsyncOpenAI()
def event(kind, content):
return f"data: {json.dumps({'type': kind, 'content': content})}\n\n"
async def agent_stream(prompt):
# 1. Start LLM stream
stream = await client.chat.completions.create(
model="gpt-4.1-mini",
messages=[{"role": "user", "content": prompt}],
stream=True,
)
pending = "" # text received but not yet sent to the user
in_tool_mode = False
async for chunk in stream:
if not chunk.choices:
continue
pending += chunk.choices[0].delta.content or "" # content can be None
# 2. Detect tool blocks and never forward their contents
if not in_tool_mode and "<tool>" in pending:
before, pending = pending.split("<tool>", 1)
if before:
yield event("text", before)
in_tool_mode = True
yield event("status", "Thinking...")
if in_tool_mode:
if "</tool>" not in pending:
continue
pending = pending.split("</tool>", 1)[1]
in_tool_mode = False
# 3. Hold back a possible partial "<tool>" tag; stream everything else
cut = pending.rfind("<")
if cut != -1 and "<tool>".startswith(pending[cut:]):
safe, pending = pending[:cut], pending[cut:]
else:
safe, pending = pending, ""
if safe:
yield event("text", safe)
if pending and not in_tool_mode:
yield event("text", pending)
With native function calling the job is easier: tool calls arrive in delta.tool_calls, separate from delta.content, so you forward content and never forward tool-call deltas. The tag parsing above is for text-based (ReAct-style) agents.
4.2 The Client Consumer (React)
Using fetch with a ReadableStream.
const response = await fetch('/api/agent');
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
// Parse "event: text\ndata: hello"
handleSSE(chunk);
}
5. Code Examples: FastAPI with SSE
Here is a minimal backend using the SSE wire format.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json
import asyncio
app = FastAPI()
async def event_generator():
"""
Yields data in strict SSE format:
data: {"key": "value"}\n\n
"""
# Phase 1: Planning
yield f"data: {json.dumps({'type': 'status', 'content': 'Searching Google...'})}\n\n"
await asyncio.sleep(1) # Fake tool latency
# Phase 2: Streaming Answer
sentence = "The speed of light is 299,792 km/s."
for word in sentence.split():
yield f"data: {json.dumps({'type': 'token', 'content': word + ' '})}\n\n"
await asyncio.sleep(0.1)
# Phase 3: Done
yield "data: [DONE]\n\n"
@app.get("/stream")
async def stream_endpoint():
return StreamingResponse(event_generator(), media_type="text/event-stream")
6. Production Considerations
6.1 Buffering at the Edge
Reverse proxies and CDNs (Nginx by default, and some Cloudflare and gzip configurations) “Buffer” responses, for example to compress them. They might wait until a chunk of data has accumulated before sending anything to the user. Fix:
- Set header
X-Accel-Buffering: no(Nginx). - Set header
Cache-Control: no-cache. - Disable Gzip for
/streamendpoints.
6.2 Timeouts
Many proxies close idle connections after 60s by default (AWS ALB’s idle timeout, Nginx’s proxy_read_timeout). An agent might take 2 minutes.
Fix: Raise the idle timeout to 300s+ for streaming paths, and send an SSE heartbeat comment (: ping) every 15 to 30 seconds so the connection never looks idle.
7. Common Pitfalls
- JSON Truncation: Trying to parse a JSON object via
json.loads()while it’s still streaming (incomplete).- Fix: Use a streaming JSON parser (like
json-streamlibrary) or only parse line-delimited chunks.
- Fix: Use a streaming JSON parser (like
- Layout Shift: Streaming tokens causes the layout to shift violently (Cumulative Layout Shift).
- Fix: Set a minimum height for the chat container.
8. Best Practices: “Skeleton Loaders” for Thoughts
Don’t just stream text. Stream Status Updates. Users love to see the “Brain”:
[Status: Reading PDF...][Status: calculating...][Text: The answer is 42]
This “Transparency” reduces perceived latency.
9. Connections to Other Topics
- Streaming speech recognition has the same trade-off. Partial transcripts (
he->hel->hello) appear fast but can be revised, just as streamed agent output can be. The UX question is identical: “Stability vs Speed”. - Real-Time Agent Pipelines covers backpressure and pipeline design behind the stream.
- Cost Management for Agents: streaming does not change token cost, so long streamed answers still need budgets.
10. Real-World Examples
- Perplexity.ai: A well-known example. Sources appear one by one, then the answer streams.
- Vercel AI SDK: A library that standardizes a stream protocol for text, tool calls and status data, making it easy to hook a Next.js frontend to model provider streams.
11. Future Directions
- Generative UI: Streaming not just text, but React Components.
- Agent streams:
<WeatherWidget temp="72" />. - Browser renders the widget instantly.
- Agent streams:
- Speech-to-Speech: Streaming audio in, streaming audio out, with no separate text step (for example OpenAI’s Realtime API). Full-duplex models such as Kyutai’s Moshi go further and can listen while they speak.
12. Key Takeaways
- SSE > WebSockets: For most chat agents, SSE is simpler and friendlier to firewalls.
- Edge Buffering is the Enemy: If streaming isn’t working, check your Nginx config.
- Visual Latency: Show something (a status event or the first token) within a few hundred milliseconds, even if the model’s TTFT is longer.
- Leakage Control: Build a state machine to hide raw “Tool Calls” from the end user.
Next: Dependency Graphs for Agents covers how to model agent plans as DAGs for parallel execution.
FAQ
Should I use WebSockets or SSE for streaming AI agents?
For most chat-style AI agents, Server-Sent Events (SSE) is the better choice. SSE is simpler to implement, works with standard load balancers, and is firewall-friendly. WebSockets are bidirectional and better suited for real-time gaming or collaborative editing, but they add unnecessary complexity for one-way LLM token streaming.
What is leakage control in streaming agents?
Leakage control prevents raw internal agent data (like JSON tool call syntax, intermediate reasoning steps, or observation results) from being streamed directly to the user. You build a state machine that detects tool usage in the token stream and only forwards final answer tokens to the frontend.
Why does my SSE streaming not work in production?
Reverse proxies and CDNs (Nginx by default, and some Cloudflare and gzip configurations) buffer responses, waiting until a chunk of data accumulates before sending. Fix this by setting X-Accel-Buffering: no for Nginx, Cache-Control: no-cache headers, and disabling gzip on streaming endpoints.
What is Time to First Token (TTFT) and why does it matter?
TTFT is the delay between the user sending a request and seeing the first token of the response. For streaming agents, aim to show something (a status event or the first token) within a few hundred milliseconds. Users perceive blank screens as broken; even a brief status update like “Thinking…” makes the experience feel responsive while the agent processes.