Tool Use on RoTBench Multi-turn
72.9Tool Selection AccuracyPA-Tool
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PA-ToolBackbone=Gemini-2.5-Flash, Model Category=Large Language Models2025.10 | 72.9 | 64.3 | |
| BaseModel=GPT-4.1-mini2025.10 | 71.4 | 61.4 | |
| GPT-4.1-miniMethod=Base, Model Category=Large Language Models2025.10 | 71.4 | 61.4 | |
| PA-ToolBackbone=GPT-4.1-mini, Model Category=Large Language Models2025.10 | 70 | 61.4 | |
| Gemini-2.5-Flash-LiteMethod=Base, Model Category=Small Language Models2025.10 | 65.7 | 58.6 | |
| PA-ToolBackbone=Qwen3-4B, Model Category=Reasoning Models, Mode=thinking mode2025.10 | 65.7 | 61.4 | |
| BaseModel=Claude-Sonnet-4.52025.10 | 64.3 | 58.6 | |
| PA-ToolBackbone=Gemini-2.5-Flash-Lite, Model Category=Small Language Models2025.10 | 64.3 | 58.6 | |
| PA-ToolBackbone=Llama3.3-70B, Model Category=Large Language Models2025.10 | 64.3 | 48.6 | |
| BaseModel=Gemini-2.5-Flash2025.10 | 60 | 54.3 | |
| Llama3.3-70BMethod=Base, Model Category=Large Language Models2025.10 | 60 | 47.1 | |
| Gemini-2.5-FlashMethod=Base, Model Category=Large Language Models2025.10 | 60 | 54.3 | |
| Qwen3-4BMethod=Base, Model Category=Reasoning Models, Mode=thinking mode2025.10 | 57.1 | 52.9 | |
| Ministral-8BMethod=Base, Model Category=Small Language Models2025.10 | 51.4 | 38.6 | |
| PA-ToolBackbone=Ministral-8B, Model Category=Small Language Models2025.10 | 51.4 | 41.4 | |
| GPT-4.1-nanoMethod=Base, Model Category=Small Language Models2025.10 | 51.4 | 44.3 | |
| PA-ToolModel=Llama3.1-8B2025.10 | 48.6 | 35.7 | |
| PA-ToolBackbone=GPT-4.1-nano, Model Category=Small Language Models2025.10 | 47.1 | 42.9 | |
| MostFreqModel=Llama3.1-8B2025.10 | 45.7 | 34.3 | |
| PA-ToolBackbone=Qwen3-1.7B, Model Category=Reasoning Models, Mode=thinking mode2025.10 | 45.7 | 38.6 | |
| GreedyModel=Llama3.1-8B2025.10 | 44.3 | 32.9 | |
| BaseModel=Llama3.1-8B2025.10 | 42.8 | 34.3 | |
| Qwen3-1.7BMethod=Base, Model Category=Reasoning Models, Mode=thinking mode2025.10 | 38.6 | 38.6 | |
| PA-ToolModel=Llama3.2-3B2025.10 | 34.3 | 28.6 | |
| BaseModel=Llama3.2-3B2025.10 | 32.9 | 27.1 | |
| GreedyModel=Llama3.2-3B2025.10 | 32.9 | 25.7 | |
| MostFreqModel=Llama3.2-3B2025.10 | 32.9 | 27.1 | |
| PA-ToolModel=Qwen2.5-7B2025.10 | 27.1 | 22.9 | |
| MostFreqModel=Qwen2.5-7B2025.10 | 24.3 | 20 | |
| BaseModel=Qwen2.5-7B2025.10 | 21.4 | 21.4 | |
| GreedyModel=Qwen2.5-7B2025.10 | 18.6 | 15.7 | |
| PA-ToolModel=Qwen2.5-3B2025.10 | 15.7 | 14.3 | |
| MostFreqModel=Qwen2.5-3B2025.10 | 12.9 | 8.6 | |
| GreedyModel=Qwen2.5-3B2025.10 | 11.4 | 7.1 | |
| BaseModel=Qwen2.5-3B2025.10 | 10 | 10 |