Multi-hop tool use on ToolHop
53.97Answer CorrectnessClaude 4.0-Sonnet
Evaluation Results
| Method | Links | |
|---|---|---|
| Claude 4.0-SonnetVersion=4.0-Sonnet2025.08 | 53.97 | |
| Qwen3-32B ReasoningFamily=Qwen3-32B, Version=Reasoning2025.08 | 50.12 | |
| PACTBackbone=Qwen3-4B2026.06 | 49.55 | |
| MatchTIRBackbone=Qwen3-4B2026.06 | 47.54 | |
| SFT→MatchTIRBackbone=Qwen3-4B2026.06 | 47.34 | |
| MatchTIR (KM)Backbone=Qwen3-8B2026.01 | 46.16 | |
| MatchTIR (OT)Backbone=Qwen3-8B2026.01 | 45.8 | |
| Gemini 2.5-ProVersion=2.5-Pro2025.08 | 45.32 | |
| Qwen3-14B Reasoning FTRL-Reinforce++Family=Qwen3-14B, Version=Reasoning, Method=FTRL-Reinforce++2025.08 | 44.79 | |
| FTRL-MBackbone=Qwen3-8B2026.01 | 43.32 | |
| FTRLBackbone=Qwen3-4B2026.06 | 43.02 | |
| MatchTIR (KM)Backbone=Qwen3-4B2026.01 | 42.55 | |
| ToolRL-MBackbone=Qwen3-8B2026.01 | 42.55 | |
| VanillaBackbone=Qwen3-8B2026.01 | 42.21 | |
| MatchTIR (OT)Backbone=Qwen3-4B2026.01 | 41.95 | |
| Qwen3-14B Reasoning FTRL-GRPOFamily=Qwen3-14B, Version=Reasoning, Method=FTRL-GRPO2025.08 | 41.64 | |
| FTRL-MBackbone=Qwen3-4B2026.01 | 41.24 | |
| Qwen3-8B Reasoning FTRL-Reinforce++Family=Qwen3-8B, Version=Reasoning, Method=FTRL-Reinforce++2025.08 | 41.24 | |
| Qwen3-8B ReasoningFamily=Qwen3-8B, Version=Reasoning2025.08 | 40.7 | |
| GRPOBackbone=Qwen3-8B2026.01 | 40.64 | |
| Qwen3-8B Reasoning FTRL-GRPOFamily=Qwen3-8B, Version=Reasoning, Method=FTRL-GRPO2025.08 | 40.5 | |
| Qwen3-14B ReasoningFamily=Qwen3-14B, Version=Reasoning2025.08 | 40.1 | |
| Gemini 2.5-FlashVersion=2.5-Flash2025.08 | 39.37 | |
| Qwen3-8B Non-Reasoning FTRL-GRPOFamily=Qwen3-8B, Version=Non-Reasoning, Method=FTRL-GRPO2025.08 | 38.93 | |
| FTRL-SBackbone=Qwen3-4B2026.01 | 38.63 | |
| Qwen3-14B Non-Reasoning FTRL-Reinforce++Family=Qwen3-14B, Version=Non-Reasoning, Method=FTRL-Reinforce++2025.08 | 38.32 | |
| Qwen3-14B Non-Reasoning FTRL-GRPOFamily=Qwen3-14B, Version=Non-Reasoning, Method=FTRL-GRPO2025.08 | 38.26 | |
| Qwen3-8B Non-Reasoning FTRL-Reinforce++Family=Qwen3-8B, Version=Non-Reasoning, Method=FTRL-Reinforce++2025.08 | 37.79 | |
| CHORDBackbone=Qwen3-4B2026.06 | 37.69 | |
| GRPOBackbone=Qwen3-4B2026.01 | 37.25 | |
| FTRL-SBackbone=Qwen3-8B2026.01 | 36.72 | |
| Qwen2.5-14B FTRL-Reinforce++Family=Qwen2.5-14B, Method=FTRL-Reinforce++2025.08 | 36.42 | |
| GPT 4oVersion=4o2025.08 | 36.31 | |
| ToolRL-MBackbone=Qwen3-4B2026.01 | 35.68 | |
| Qwen3-32B Non-ReasoningFamily=Qwen3-32B, Version=Non-Reasoning2025.08 | 35.34 | |
| PACTBackbone=Qwen3-1.7B2026.06 | 34.47 | |
| GRPOBackbone=Qwen3-4B2026.06 | 34.47 | |
| ToolRLBackbone=Qwen3-4B2026.06 | 34.27 | |
| ToolRL-SBackbone=Qwen3-8B2026.01 | 32.93 | |
| MatchTIRBackbone=Qwen3-1.7B2026.06 | 32.26 | |
| VanillaBackbone=Qwen3-4B2026.06 | 31.86 | |
| VanillaBackbone=Qwen3-4B2026.01 | 31.63 | |
| SFT→MatchTIRBackbone=Qwen3-1.7B2026.06 | 31.06 | |
| Qwen2.5-72B InstructFamily=Qwen2.5-72B, Version=Instruct2025.08 | 30.29 | |
| ToolRL-SBackbone=Qwen3-4B2026.01 | 30.28 | |
| SFTBackbone=Qwen3-4B2026.06 | 30.25 | |
| Qwen2.5-7B FTRL-GRPOFamily=Qwen2.5-7B, Method=FTRL-GRPO2025.08 | 29.68 | |
| Qwen3-8B Non-ReasoningFamily=Qwen3-8B, Version=Non-Reasoning2025.08 | 28.54 | |
| CHORDBackbone=Qwen3-1.7B2026.06 | 27.14 | |
| Qwen2.5-14B FTRL-GRPOFamily=Qwen2.5-14B, Method=FTRL-GRPO2025.08 | 26.63 | |
| FTRLBackbone=Qwen3-1.7B2026.06 | 26.43 | |
| ToolRLBackbone=Qwen3-1.7B2026.06 | 25.83 | |
| Qwen2.5-14B InstructFamily=Qwen2.5-14B, Version=Instruct2025.08 | 25.8 | |
| GPT 3.5-TurboVersion=3.5-Turbo2025.08 | 25.73 | |
| VanillaBackbone=Qwen3-1.7B2026.06 | 25.63 | |
| GRPOBackbone=Qwen3-1.7B2026.06 | 25.13 | |
| Qwen3-14B Non-ReasoningFamily=Qwen3-14B, Version=Non-Reasoning2025.08 | 24.19 | |
| Qwen2.5-7B FTRL-Reinforce++Family=Qwen2.5-7B, Method=FTRL-Reinforce++2025.08 | 23.45 | |
| SFTBackbone=Qwen3-1.7B2026.06 | 23.32 | |
| Qwen2.5-32B InstructFamily=Qwen2.5-32B, Version=Instruct2025.08 | 19.7 | |
| Qwen2.5-7B InstructFamily=Qwen2.5-7B, Version=Instruct2025.08 | 11.99 |