Agentic Task Success on MCP-Universe
70.8Financial Success ScorePREPING
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PREPINGMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 70.8 | — | 32.3 | — | — | 37.5 | 10.1 | 36.8 | — | |
| SINGSetting=Restricted2026.06 | 70 | 28.6 | 50 | 50 | — | 47.1 | 32.1 | 42.1 | 69.5 | |
| ACE-OnlineMethod Category=Task-Informed Methods, Base LLM=DeepSeek-V3.22026.05 | 69.2 | — | 37.5 | — | — | 41.2 | 14.1 | 43.9 | — | |
| Direct MemoryMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 67.5 | — | 30.2 | — | — | 32.8 | 7.1 | 26.3 | — | |
| SINGSetting=Global2026.06 | 65 | 31.4 | 52.9 | 46 | — | 45.1 | 26.4 | — | 72.7 | |
| GPT-OSSSFT=evaluated without SFT2026.06 | 62.5 | 25.71 | — | 26 | — | 32.13 | 14.28 | — | — | |
| HyperToolBackbone=Qwen3-8B, SFT=fine-tuned on HyperTool trajectory data2026.06 | 62.5 | 28.57 | — | 18 | — | 33.33 | 25 | — | — | |
| HyperToolBackbone=Qwen3-32B, SFT=fine-tuned on HyperTool trajectory data2026.06 | 62.5 | 34.29 | — | 22 | — | 35.29 | 21.43 | — | — | |
| MCP-ZeroSetting=Global2026.06 | 62.5 | 28.6 | 35.3 | 22 | — | 35 | 25 | 36.8 | 46.7 | |
| Guided ExplorationMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 61.7 | — | 31.3 | — | — | 34.6 | 12.1 | 33.3 | — | |
| Random ExplorationMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 60.8 | — | 38.5 | — | — | 33.1 | 10.1 | 22.8 | — | |
| Ground Truth ServerSetting=Default2026.06 | 60 | 31.4 | 38.2 | 42 | — | 40.3 | 17.9 | 47.4 | 63 | |
| MCP-ZeroSetting=Restricted2026.06 | 60 | 22.9 | 41.2 | 44 | — | 39 | 28.6 | 57.9 | 51.2 | |
| BaseMethod Category=Pre-Task Methods, Base LLM=DeepSeek-V3.22026.05 | 59.2 | — | 31.3 | — | — | 32.1 | 8.1 | 29.8 | — | |
| DeepSeek-V4-FlashSFT=evaluated without SFT2026.06 | 57.5 | 29.41 | — | 45.83 | — | 44 | 39.29 | — | — | |
| GLM-5.1SFT=evaluated without SFT2026.06 | 55 | 28.57 | — | 36 | — | 36.6 | 42.86 | — | — | |
| AgentFoldBackbone=Qwen3-32B, SFT=evaluated without SFT2026.06 | 52.4 | 14.3 | — | 16 | — | 27.13 | 26.94 | — | — | |
| Kimi-k2.5SFT=evaluated without SFT2026.06 | 51.28 | 17.14 | — | 25 | — | 34.17 | 44.44 | — | — | |
| CodeActBackbone=Qwen3-32B, SFT=evaluated without SFT2026.06 | 42.5 | 14.29 | — | 20 | — | 25.49 | 25 | — | — | |
| ReCodeBackbone=Qwen3-32B, SFT=evaluated without SFT2026.06 | 40 | 11.43 | — | 6 | — | 19.6 | 25 | — | — | |
| ReActBackbone=Qwen3-32B, SFT=fine-tuned on HyperTool trajectory data2026.06 | 40 | 20 | — | 16 | — | 24.18 | 21.43 | — | — | |
| BrowseMasterBackbone=Qwen3-32B, SFT=evaluated without SFT2026.06 | 37.5 | 17.14 | — | 10 | — | 20.91 | 21.43 | — | — | |
| AWMModel Size=8B2026.02 | 35 | 8.57 | 8.82 | 0 | 5 | 11.17 | — | — | — | |
| BaseBackbone=Qwen3-32B, SFT=evaluated without SFT2026.06 | 35 | 20 | — | 4 | — | 15.69 | 3.57 | — | — | |
| AWMModel Size=14B2026.02 | 32.5 | 8.57 | 11.77 | 4 | 0 | 12.29 | — | — | — | |
| ReActBackbone=Qwen3-8B, SFT=fine-tuned on HyperTool trajectory data2026.06 | 32.5 | 25.71 | — | 16 | — | 20.92 | 7.14 | — | — | |
| SimulatorModel Size=14B2026.02 | 30 | 2.86 | 8.82 | 6 | 0 | 10.62 | — | — | — | |
| AgentFoldBackbone=Qwen3-8B, SFT=evaluated without SFT2026.06 | 27.78 | 25.71 | — | 16 | — | 21.63 | 17.81 | — | — | |
| BaseModel Size=14B2026.02 | 27.5 | 0 | 5.88 | 2 | 5 | 8.38 | — | — | — | |
| BaseModel Size=4B2026.02 | 25 | 2.86 | 0 | 0 | 0 | 6.15 | — | — | — | |
| Gemini-2.5-FlashSFT=evaluated without SFT2026.06 | 25 | 30.77 | — | 19.51 | — | 25.58 | 30.77 | — | — | |
| ReCodeBackbone=Qwen3-8B, SFT=evaluated without SFT2026.06 | 25 | 14.29 | — | 10 | — | 15.03 | 10.71 | — | — | |
| AWMModel Size=4B2026.02 | 22.5 | 0 | 2.94 | 2 | 5 | 6.7 | — | — | — | |
| BaseModel Size=8B2026.02 | 22.5 | 0 | 5.88 | 2 | 0 | 6.7 | — | — | — | |
| BrowseMasterBackbone=Qwen3-8B, SFT=evaluated without SFT2026.06 | 21.5 | 11.43 | — | 4 | — | 18.6 | 14.29 | — | — | |
| CodeActBackbone=Qwen3-8B, SFT=evaluated without SFT2026.06 | 20 | 22.86 | — | 22 | — | 20.92 | 14.29 | — | — | |
| EnvScalerModel Size=4B2026.02 | 17.5 | 0 | 2.94 | 0 | 0 | 4.47 | — | — | — | |
| EnvScalerModel Size=8B2026.02 | 17.5 | 0 | 5.88 | 2 | 0 | 5.59 | — | — | — | |
| BaseBackbone=Qwen3-8B, SFT=evaluated without SFT2026.06 | 17.14 | 17.14 | — | 4 | — | 9.93 | 3.57 | — | — | |
| SimulatorModel Size=4B2026.02 | 15 | 0 | 5.88 | 2 | 0 | 6.15 | — | — | — | |
| SimulatorModel Size=8B2026.02 | 10 | 5.71 | 8.82 | 4 | 0 | 6.15 | — | — | — |