FinMCP: why financial AI agents need their own benchmark
Your agent scores 87% on GAIA and 73% on WebArena. You deploy it to handle insurance underwriting queries. It fails at 40% of real tasks. The benchmarks told you something true. They just didn’t tell you the right thing.
This is the core problem with evaluating production agents on general benchmarks. They measure whether a model has broad capability. They don’t measure whether it will work in your specific environment, under your specific constraints, with your specific tools. The gap between those two things is where deployments go wrong.
FinMCP-Bench, published March 26, 2026 (arXiv:2603.24943), makes this concrete for financial agents. It’s a benchmark built specifically for LLM agents using Model Context Protocol tools in real financial workflows. The numbers aren’t surprising so much as clarifying: they name the failure modes precisely, which is exactly what general benchmarks don’t do.
TL;DR: FinMCP-Bench tests LLM agents on 613 financial tasks (real and synthetic queries) using 65 real financial MCP tools. The best model reaches 64.3% tool F1 overall, but its exact match rate falls to 10.6% on multi-tool tasks and 3.1% on multi-turn ones. Generic benchmarks don’t surface this. For financial, legal, and medical agents, domain-specific evals aren’t optional. They’re the only signal that matters.
What generic benchmarks can’t measure
The best-scoring model on FinMCP-Bench achieves 64.27% tool F1 overall. On multi-tool tasks, its exact match rate is 10.62%, and the six models average about 5.4%. Multi-turn tasks are harder still: exact match ranges from 0% to 4.1%. No general benchmark tracks this specific collapse. GAIA, WebArena, and most agent evals measure whether a model can navigate open-ended tasks, use web tools, and reason across multiple steps. What they don’t test is what happens when those steps must occur in a specific order mandated by regulation, not by logic.
FinMCP’s exact match metric does not test regulation directly; it checks whether the agent’s tool calls match the reference workflow. That is the right proxy, because in financial workflows the order of tool calls is often legally prescribed. You fetch compliance eligibility before executing an allocation recommendation. You verify regulatory status before generating a prospectus summary. You resolve jurisdiction before applying tax treatment. A model that gets the right final answer but in the wrong sequence has produced a compliance violation, not a success.
Generic benchmarks also don’t test arithmetic precision under latency. Financial computations (NAV calculations, margin requirements, yield spreads) require exact numeric results. A model that rounds, approximates, or uses cached values from an earlier tool call introduces errors that compound across a multi-step workflow. Standard evals reward “approximately correct.” Financial production cannot.
The consequence is predictable: agents that look capable on general benchmarks often aren’t capable at the thing you actually need them to do.
Related: Agent evaluation frameworks covers the broader evaluation landscape.
What FinMCP-Bench actually tests
The benchmark exposes exactly the failure modes that matter for financial deployment. FinMCP-Bench contains 613 samples across 10 main financial scenarios and 33 subcategories, built using 65 real financial MCP tools. The two largest scenarios are market analysis and research (141 samples) and investment planning and allocation (101 samples).
Three task types capture different levels of complexity. Single-tool tasks (145 samples) require one tool call to answer a query. Multi-tool tasks (249 samples) require an average of 7.32 tool calls across 5.72 steps, some of them parallel, coordinating across data sources in a defined order. Multi-turn tasks (219 samples) test conversational coherence across an average of 5.95 dialogue turns, where the agent must maintain context and update its reasoning as new information arrives.
The multi-tool category is where financial realism lives. Real investment workflows don’t call one API. They chain market data retrieval, portfolio constraint checks, compliance validation, and allocation execution, each step dependent on the last. A model that can pick the right tool for a single query but can’t sequence them correctly under constraint isn’t useful in production.
The benchmark also incorporates both real user queries from financial practitioners and synthetic queries designed to stress-test edge cases. That combination matters: synthetic queries alone don’t capture the ambiguity and domain vocabulary of real user requests, but real queries alone don’t systematically cover the failure modes you need to test.
How current agents score on FinMCP
Six models were evaluated, and the spread is wider than you’d expect from their general benchmark rankings. The top model, Qwen3-235B-A22B-Thinking, scores 64.27% overall tool F1. Qwen3-30B-A3B-Thinking comes in at 55.58%, Qwen3-4B-Thinking at 50.08%, DeepSeek-R1 at 49.88%, Seed-OSS-36B at 39.34%, and GPT-OSS-20B at 32.62%.
That’s a 32-percentage-point spread between the best and worst models, and the ranking does not follow model size: the 4B Qwen3 model edges out the 671B DeepSeek-R1. Domain benchmarks reveal differentiation that general evals smooth over.
The multi-tool collapse is the most important finding. Exact match rates on multi-tool tasks run from 1.2% (GPT-OSS-20B) to 10.6% (Qwen3-235B), about 5.4% on average. Even the best model gets the full tool organization right on only about 1 task in 10. The models can identify relevant tools. They just can’t reliably orchestrate them in the order the workflow requires.
Performance is also asymmetric across task types. Single-tool exact match reaches 49% to 66% for most models, and the paper notes that gaps between models narrow on simpler queries and widen in multi-tool planning. Multi-turn conversations, where the agent must track state across a dialogue, produce the lowest exact match rates of any task type. The models are better at retrieval than at stateful reasoning under constraint.
This doesn’t mean current models are useless for financial workflows. It means the failure modes are specific and testable. That’s exactly what a good benchmark gives you.
Extending the pattern: domain benchmarks for legal, medical, and infrastructure
FinMCP’s methodology is a template, not a one-off. The question it asks is worth asking in every domain that has real stakes: what does failure actually look like here, and can our benchmark catch it?
In legal, the answer is different from finance but the gap is similar. On Vals AI’s LegalBench leaderboard, built on the crowd-sourced legal reasoning benchmark, top models scored about 87% in early 2026 (Gemini 3 Pro Preview at 87.0%). That measures general legal reasoning. It doesn’t tell you whether an agent can draft a compliant NDA under Delaware law using a law firm’s specific document management tools, in the right format, with correct citations. LegalBenchmarks.ai is building toward that, but the field is early. The failure modes in legal are jurisdiction specificity and output format compliance, things that look fine in a generic eval and surface immediately in a real client engagement.
Medical is closer to where finance is now. Stanford’s MedAgentBench evaluates agents on 300 clinical tasks inside a simulated, FHIR-compliant EHR with realistic records for 100 patients: ordering medications, retrieving patient data, coordinating care decisions. The insight is the same one FinMCP demonstrates: performing well on USMLE-style knowledge questions doesn’t predict whether a model can execute clinical workflows correctly inside a real system with real data and real consequence for error. Knowledge and workflow fidelity are different things.
Infrastructure agents face a problem that’s less about compliance and more about idempotency. An agent that calls a deployment API twice because it didn’t correctly track whether the first call succeeded creates a real incident. Generic benchmarks don’t test for double-execution. Neither do most internal eval suites, until something breaks in production.
Here’s the template FinMCP implicitly demonstrates for building a domain benchmark:
flowchart TD
A[Identify domain failure modes] --> B[Collect real tools from production]
B --> C[Build samples at 3 complexity levels]
C --> D1[Single-step: tool selection accuracy]
C --> D2[Multi-step: ordering under constraint]
C --> D3[Multi-turn: stateful coherence]
D1 --> E[Measure both Tool F1 and Exact Match Rate]
D2 --> E
D3 --> E
E --> F[Find the models that don't collapse on your hard cases]
The critical step is identifying the failure modes first. What does your agent do that, if wrong, produces a compliance violation, a safety incident, or a regulatory penalty? Start there. Everything else flows from it.
Score gap, from the paper: overall tool F1 hides how badly sequencing degrades.
Model | FinMCP tool F1 | Overall EMR | Multi-tool EMR | Multi-turn EMR
Qwen3-235B | 64.27% | 25.92% | 10.62% | 3.08%
DeepSeek-R1 | 49.88% | 18.08% | 4.85% | 0.00%
GPT-OSS-20B | 32.62% | 4.43% | 1.20% | 0.00%
All figures from arXiv:2603.24943 (Table 2).
The point isn’t which specific model performs best. It’s that a model can look reasonable on an aggregate score and still almost never reproduce a multi-step workflow exactly. Tool F1 tells you it picks plausible tools; exact match tells you whether the workflow would actually run as designed.
Building your domain benchmark: practical starting point
If you’re deploying agents in any regulated domain, here’s a concrete starting point based on FinMCP’s structure:
Step 1. List the five things your agent must never get wrong. These are your hard case seeds, the failure modes that matter in production. They are the low-tolerance actions in the commercial viability framework, and they deserve the most test cases.
Step 2. Collect the real tools your agent uses. Don’t simulate them. FinMCP uses 65 real financial MCPs because synthetic tool behavior doesn’t capture the error states and response formats that production tools produce.
Step 3. Build samples at three levels: single-tool (test tool selection), multi-tool (test sequencing under constraint), and multi-turn (test stateful coherence). Aim for a roughly 25/40/35 split, similar to FinMCP’s distribution.
Step 4. Measure two things separately: Tool F1 (does the model call the right tools?) and Exact Match Rate (does its whole tool sequence match the reference, in the right order and grouping?). The gap between those two numbers tells you whether you have a selection problem or a sequencing problem.
Step 5. Run your benchmark before selecting a model for production, not after. The variation across models on domain-specific tasks is large enough to change your deployment decision.
Related: Benchmarking AI agents in depth and testing AI agents in production.
FAQ
What is FinMCP-Bench? FinMCP-Bench (arXiv:2603.24943) is a benchmark for evaluating LLM agents on real-world financial tasks via the Model Context Protocol. It has 613 samples across 10 financial scenarios using 65 real MCP tools, covering single-tool, multi-tool, and multi-turn task types. Published March 26, 2026, by researchers from Alibaba Cloud’s Qwen DianJin Team, YINGMI Wealth Management, and Soochow University.
How do models perform on FinMCP vs general benchmarks? The best model (Qwen3-235B-A22B-Thinking) scores 64.27% tool F1 on FinMCP overall. On multi-tool tasks its exact match rate is 10.62%, and the six-model average is about 5.4%. General benchmarks don’t expose this failure mode because they don’t test exact tool sequencing.
What financial tasks does FinMCP-Bench cover? Ten main scenarios with 33 subcategories. The two largest are market analysis and research (141 samples) and investment planning and allocation (101 samples).
Why do agents collapse on multi-tool financial sequences? Multi-tool tasks average 7.32 tool calls over 5.72 steps, and exact match requires the whole tool organization to match the reference workflow. In production finance, a right answer reached in the wrong sequence can be a compliance violation, not a success. Generic benchmarks don’t test ordering under constraint; they reward approximate correctness.
How do you build a domain-specific benchmark for other fields? FinMCP’s method: identify the irreducible failure modes in your domain first, collect real tools from production environments, build samples at three complexity levels (single-step, multi-step, multi-turn), measure tool selection and execution order separately. Legal and medical domains need the same approach. The pattern is transferable.