Tool Selection on MetaTool
80.8SimilarityPA-Tool
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PA-ToolBackbone=Gemini-2.5-Flash-Lite, Model Category=Small Language Models2025.10 | 80.8 | 87.1 | 70.5 | 72 | — | — | — | — | — | |
| PA-ToolBackbone=GPT-4.1-mini, Model Category=Large Language Models2025.10 | 80.5 | 84.5 | 80 | 84.3 | — | — | — | — | — | |
| Llama3.3-70BMethod=Base, Model Category=Large Language Models2025.10 | 80.4 | 85.2 | 63.7 | 82.9 | — | — | — | — | — | |
| PA-ToolBackbone=Llama3.3-70B, Model Category=Large Language Models2025.10 | 79.9 | 85.2 | 66.2 | 81.3 | — | — | — | — | — | |
| BaseModel=GPT-4.1-mini2025.10 | 79.6 | 84.3 | 76.3 | 72.2 | — | — | — | — | — | |
| GPT-4.1-miniMethod=Base, Model Category=Large Language Models2025.10 | 79.6 | 84.3 | 76.3 | 72.2 | — | — | — | — | — | |
| Gemini-2.5-Flash-LiteMethod=Base, Model Category=Small Language Models2025.10 | 79.3 | 86.5 | 65.9 | 77.7 | — | — | — | — | — | |
| BaseModel=Claude-Sonnet-4.52025.10 | 75.6 | 83 | 84.3 | 85.1 | — | — | — | — | — | |
| PA-ToolBackbone=Qwen3-4B, Model Category=Reasoning Models, Mode=thinking mode2025.10 | 73.5 | 81.2 | 91.2 | 86.5 | — | — | — | — | — | |
| PA-ToolBackbone=Gemini-2.5-Flash, Model Category=Large Language Models2025.10 | 72.6 | 82.7 | 92.4 | 78.5 | — | — | — | — | — | |
| Qwen3-4BMethod=Base, Model Category=Reasoning Models, Mode=thinking mode2025.10 | 72 | 81.9 | 84.2 | 88.1 | — | — | — | — | — | |
| PA-ToolBackbone=Ministral-8B, Model Category=Small Language Models2025.10 | 70.5 | 82.6 | 44.7 | 92.4 | — | — | — | — | — | |
| PA-ToolModel=Llama3.1-8B2025.10 | 70.4 | 79.9 | 66 | 88.3 | — | — | — | — | — | |
| BaseModel=Gemini-2.5-Flash2025.10 | 70 | 79.8 | 89.2 | 77.3 | — | — | — | — | — | |
| Gemini-2.5-FlashMethod=Base, Model Category=Large Language Models2025.10 | 70 | 79.8 | 89.2 | 77.3 | — | — | — | — | — | |
| HumanModel=Llama3.1-8B2025.10 | 69.3 | 78.9 | 63.5 | 86.5 | — | — | — | — | — | |
| MostFreqModel=Llama3.1-8B2025.10 | 68.8 | 79.3 | 66.4 | 85.7 | — | — | — | — | — | |
| PA-ToolBackbone=Qwen3-1.7B, Model Category=Reasoning Models, Mode=thinking mode2025.10 | 68.7 | 69.8 | 96.6 | 73.2 | — | — | — | — | — | |
| Qwen3-1.7BMethod=Base, Model Category=Reasoning Models, Mode=thinking mode2025.10 | 66.2 | 73.2 | 93.2 | 76.1 | — | — | — | — | — | |
| HumanModel=Qwen2.5-7B2025.10 | 65.8 | 76.7 | 78.6 | 82.1 | — | — | — | — | — | |
| PA-ToolModel=Llama3.2-3B2025.10 | 65.7 | 67.7 | 60.6 | 80.5 | — | — | — | — | — | |
| MostFreqModel=Qwen2.5-7B2025.10 | 65.3 | 75.5 | 90 | 86.1 | — | — | — | — | — | |
| Ministral-8BMethod=Base, Model Category=Small Language Models2025.10 | 65.3 | 81.2 | 42.5 | 93.1 | — | — | — | — | — | |
| GreedyModel=Llama3.1-8B2025.10 | 64.6 | 72.9 | 51.5 | 78.9 | — | — | — | — | — | |
| PA-ToolModel=Qwen2.5-7B2025.10 | 64.1 | 78.4 | 88.2 | 84.9 | — | — | — | — | — | |
| BaseModel=Llama3.1-8B2025.10 | 61.5 | 73.9 | 53.5 | 78.7 | — | — | — | — | — | |
| MostFreqModel=Llama3.2-3B2025.10 | 61.3 | 64.2 | 65.7 | 81.3 | — | — | — | — | — | |
| GreedyModel=Qwen2.5-7B2025.10 | 60.6 | 75.4 | 85.4 | 82.7 | — | — | — | — | — | |
| PA-ToolBackbone=GPT-4.1-nano, Model Category=Small Language Models2025.10 | 60.6 | 54 | 99.5 | 77.1 | — | — | — | — | — | |
| BaseModel=Qwen2.5-7B2025.10 | 59.6 | 74.4 | 78.3 | 78.3 | — | — | — | — | — | |
| HumanModel=Llama3.2-3B2025.10 | 58.6 | 60.9 | 55.2 | 72.6 | — | — | — | — | — | |
| GreedyModel=Llama3.2-3B2025.10 | 57.7 | 58.9 | 39.8 | 70.4 | — | — | — | — | — | |
| GPT-4.1-nanoMethod=Base, Model Category=Small Language Models2025.10 | 56.8 | 53.1 | 99 | 81.5 | — | — | — | — | — | |
| BaseModel=Llama3.2-3B2025.10 | 55 | 58.6 | 43.6 | 79.1 | — | — | — | — | — | |
| HumanModel=Qwen2.5-3B2025.10 | 54.6 | 62.9 | 80.6 | 72.6 | — | — | — | — | — | |
| MostFreqModel=Qwen2.5-3B2025.10 | 50.5 | 56.3 | 84.3 | 73 | — | — | — | — | — | |
| PA-ToolModel=Qwen2.5-3B2025.10 | 50 | 58.8 | 86.2 | 72.6 | — | — | — | — | — | |
| GreedyModel=Qwen2.5-3B2025.10 | 49.1 | 57 | 82.9 | 63.4 | — | — | — | — | — | |
| BaseModel=Qwen2.5-3B2025.10 | 48.7 | 55.3 | 83.6 | 75.1 | — | — | — | — | — | |
| Best Fixed-KScorer=BM25, N=1002026.05 | — | — | — | — | — | — | 83.7 | — | — | |
| Best Fixed-KScorer=embed, N=1002026.05 | — | — | — | — | — | — | 74.7 | — | — | |
| Best Fixed-KScorer=BGE, N=1002026.05 | — | — | — | — | — | — | 57.2 | — | — | |
| Best Fixed-KN=202026.05 | — | — | — | — | — | — | 59.2 | — | — | |
| Best Fixed-KN=502026.05 | — | — | — | — | — | — | 59.2 | — | — | |
| Best Fixed-KN=1002026.05 | — | — | — | — | — | — | 59.2 | — | — | |
| BoRScorer=BM25, N=1002026.05 | — | — | — | — | 96.2 | — | — | 80.7 | — | |
| BoRScorer=embed, N=1002026.05 | — | — | — | — | 73.3 | — | — | 2.3 | — | |
| BoRScorer=BGE, N=1002026.05 | — | — | — | — | 71.4 | — | — | 2.4 | — | |
| BoRN=202026.05 | — | — | — | — | 70.6 | — | — | 1.6 | — | |
| BoRN=502026.05 | — | — | — | — | 74.7 | — | — | 2.6 | — | |
| BoRN=1002026.05 | — | — | — | — | 72.8 | — | — | 2.1 | — | |
| F1Scorer=BM25, N=1002026.05 | — | — | — | — | — | 82.8 | — | — | 57.2 | |
| F1Scorer=embed, N=1002026.05 | — | — | — | — | — | 69 | — | — | 3 | |
| F1Scorer=BGE, N=1002026.05 | — | — | — | — | — | 61.4 | — | — | 1.2 | |
| F1N=202026.05 | — | — | — | — | — | 63.3 | — | — | 1.3 | |
| F1N=502026.05 | — | — | — | — | — | 63.9 | — | — | 1.2 | |
| F1N=1002026.05 | — | — | — | — | — | 63.3 | — | — | 1.5 |