Tool Use Evaluation on ShortcutsBench clear instructions 200 queries
92.5API Selection AccuracyGPT-4o
Evaluation results
| Method | Links | ||
|---|---|---|---|
| GPT-4oFramework=AwN2024.08 | 92.5 | 42.5 | |
| DeepSeek V3Framework=AwN2024.08 | 91.5 | 53.5 | |
| DeepSeek V3Framework=Base Model2024.08 | 89 | 53.5 | |
| Claude 3.5Framework=Base Model2024.08 | 89 | 46 | |
| Claude 3.5Framework=AwN2024.08 | 88.5 | 50 | |
| Gemini 1.5Framework=AwN2024.08 | 87 | 46 | |
| GPT-4oFramework=Base Model2024.08 | 86 | 45 | |
| Gemini 1.5Framework=Base Model2024.08 | 83.5 | 46 |