Is chain-of-thought a mirage? What VisBrowse-Bench reveals about reasoning limits
“Remove the images from your multimodal reasoning chain. If accuracy drops less than 5%, your agent is not actually looking.”
TL;DR
Chain-of-thought works for text. It struggles with images. VisBrowse-Bench (169 visual QA instances for browsing agents) shows Claude-4.6-Opus at 47.6% and o3-deep-research at 41.1%. The “CoT mirage” paper finds CoT reflects patterns learned from training data, not abstract reasoning. A separate study found only about a 2% accuracy drop when images were removed midway through long reasoning chains: the later reasoning leans on text, not visual evidence. For the latent-space reasoning approach that bypasses visible CoT entirely, see when LLMs stop talking to themselves.
What does VisBrowse-Bench actually measure?
Most agent benchmarks test text-based tasks: answer a question, write code, search the web. VisBrowse-Bench tests whether browsing agents can reason about what they see (images on web pages, visual layouts, screenshots), not just what they read.
The benchmark contains 169 VQA instances constructed by human experts. Each requires the agent to actively collect visual information during web search, cross-validate evidence across text and images, and reason jointly over both modalities.
Results from the paper (arXiv 2603.16289):
| Model | Accuracy |
|---|---|
| Claude-4.6-Opus | 47.6% |
| o3-deep-research | 41.1% |
Less than half. On a benchmark designed to be solvable by a human who can browse the web and look at images. The gap between text-domain reasoning (where these models routinely score 80-95%) and visual-native reasoning is enormous.
This is not a story about these models being bad. It is a data point about what happens when reasoning leaves the text domain. The same models that dominate text benchmarks struggle when the evidence is primarily visual.
Is chain-of-thought real reasoning or sophisticated mimicry?
A research team built DataAlchemy: an isolated environment to train LLMs from scratch and test reasoning under controlled distribution conditions (arXiv 2508.01191). Their finding: sophisticated CoT prompting does not endow models with abstract, generalizable inference. Models succeed on familiar problem patterns. They fail when identical logical structures appear in unfamiliar contexts.
The conclusion, stated plainly: LLMs are “sophisticated simulators of reasoning-like text” rather than principled reasoners. CoT reflects structured inductive bias learned from in-distribution training data.
This sparked considerable debate on Hacker News and Reddit. The counter-argument is philosophical: if the model produces correct answers to novel problems, does it matter whether the internal mechanism is “real” reasoning? The empirical response: the failures are not random. They cluster at distribution boundaries, exactly where pattern matching breaks and abstract reasoning would be needed.
Separately, Anthropic’s faithfulness research adds nuance. Lanham et al. (2023, arXiv 2307.13702) found large variation across tasks: sometimes corrupting the reasoning changes the answer, sometimes the model reaches the same answer regardless, which points to post-hoc rationalization. Larger models were less faithful on most tasks. Chen et al. (2025, arXiv 2505.05410) found that reasoning models often fail to mention hints that shaped their answers, with the reveal rate often below 20%. And Anthropic’s interpretability work on Claude showed that its own account of how it did mental arithmetic did not match the parallel pathways it actually used. The step-by-step narrative is at best a partial account of the computation.
Both findings matter for practitioners. CoT is useful. It improves accuracy on in-distribution tasks. It is not magic: it does not create reasoning capabilities the model lacks. And its benefits diminish sharply when the domain shifts from text to vision.
Why does CoT fail on visual inputs?
The most striking finding from recent multimodal research: when Sun et al. (arXiv 2503.13360) removed the image midway through QVQ-72B’s long reasoning chains on a hard subset of MathVista, accuracy dropped only about 2%. Past that point, the model was reasoning almost entirely from its own text, not from the image it was given.
This is the visual forgetting problem. As a reasoning chain grows longer, the model’s attention to visual embeddings decreases progressively. Language tokens occupy sequential attention positions that crowd out the image representation. Deep into a long chain, the model has effectively stopped looking at the image.
Additional failure modes compound the problem:
Low-level perception failure. Models struggle to connect shapes, sizes, and spatial layouts with semantic reasoning. They can identify objects in an image. They cannot reliably reason about the spatial relationships between those objects.
Multi-object reasoning. Basic counting and localization tasks (“how many red circles are in the image?”) produce surprising failures. These are tasks that seem trivial for a system that processes visual input.
Robustness to perturbation. Minimal image changes (small rotations, slight color shifts) can cause large performance drops.
Distractor interference. Irrelevant background elements interfere with reasoning about the foreground. Human visual reasoning filters distractors automatically. Model reasoning does not.
graph TD
A[Visual input + Text question] --> B[Step 1: Perceive image<br/>Visual attention: HIGH]
B --> C[Step 2: Begin reasoning<br/>Visual attention: MEDIUM]
C --> D[Step 3: Continue chain<br/>Visual attention: LOW]
D --> E[Step 4: Generate answer<br/>Visual attention: ~ZERO]
E --> F[Answer based mostly<br/>on text, not image]
G[Visual Forgetting:<br/>~2% accuracy drop<br/>when images removed] -.-> F
The implication for multimodal agent design is direct: if your browsing agent reasons about a web page through a long CoT chain, it is increasingly reasoning about the text it extracted from the page, not about what the page looks like. Visual layout information, image content, and spatial relationships are lost.
What alternatives exist to visible chain-of-thought?
Three approaches address the visual reasoning gap.
Decouple perception from reasoning. The IPVR framework uses a “see, think, confirm” pattern. First, extract visual facts from the image (perception step: short, focused, image-attended). Then reason over the extracted facts as text (reasoning step: can be long, no visual forgetting). Then confirm against the original image (verification step: re-anchors to visual input). This separation prevents the reasoning chain from drifting away from visual evidence.
Latent reasoning. Meta’s Coconut (Chain of Continuous Thought) reasons in the model’s internal latent space rather than generating visible text tokens. The last hidden state becomes the reasoning representation, fed back as an embedding without decoding to text. Coconut itself is text-only, so whether it helps with images is untested. It outperforms visible CoT on some logical planning tasks (ProsQA) while generating far fewer tokens, but trails CoT on math word problems. The visual counterpart is Latent Visual Reasoning (LVR, arXiv 2509.24251), which interleaves text with latent steps trained to reconstruct the image tokens most relevant to the question, so the image stays in the reasoning process.
Multimodal CoT. The two-stage rationale-then-answer framework (arXiv 2302.00923) generates a text rationale from multimodal input, then uses that rationale to produce the answer. A 1B model using this approach outperformed GPT-3.5 by 16 percentage points on ScienceQA (91.68% vs 75.17%). Separately, interleaved-modal CoT alternates between text and image reasoning steps, re-processing the image at each visual step to prevent forgetting.
For practitioners building multimodal agents, the concrete recommendations:
- Test with images removed. If accuracy drops less than 5%, your agent is not using visual evidence.
- Keep visual reasoning chains short. Three steps maximum before re-anchoring to the image.
- Decouple perception from reasoning. Extract visual facts first, then reason over the extraction as text.
- Re-anchor before critical steps. Explicitly re-process the image before any reasoning step that depends on visual content.
- Watch latent approaches. Latent visual reasoning methods such as LVR aim to keep visual information in the representation throughout; they are research results, not drop-in tools yet.
Key takeaways
- VisBrowse-Bench exposes the visual gap. Claude-Opus at 47.6%, o3-deep-research at 41.1%. Visual-native reasoning is far behind text-domain performance.
- CoT is useful, not magical. It improves in-distribution accuracy. It does not create abstract reasoning. Benefits diminish at distribution boundaries.
- Models drift away from images during long reasoning. In one study, removing the image midway through cost only about 2% accuracy. Visual forgetting is measurable and progressive.
- Decouple perception from reasoning. Extract visual facts first, then reason over text. This prevents the text-dominance bias from corrupting visual evidence.
- Latent reasoning is an emerging alternative. LVR reasons over visual embeddings directly, avoiding part of the text bottleneck of visible CoT; Coconut shows the text-only version of the idea.
Further reading
- When LLMs stop talking to themselves: latent-space reasoning and the end of visible thinking
- Multi-step reasoning: the text-domain reasoning patterns that CoT is based on
- Screenshot understanding agents: how agents process visual web content