Function Calling on BFCL v3 2025-08-26 (test)
50Multi-Turn Overall AccuracyGPT-4o-2024-11-20
Evaluation results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4o-2024-11-20Model Category=Proprietary Models2025.08 | 50 | 61 | 45.5 | 35.5 | 58 | 86.81 | 78.85 | 83.33 | 81.31 | 71.71 | |
| ToolACE-MTBackbone=Llama3.1-8B-Inst2025.08 | 40.25 | 57.5 | 31.5 | 34 | 38 | 84.94 | 71.52 | 77.78 | 72.83 | 65.41 | |
| Gemini-2.5-Pro-Preview-05-06Model Category=Proprietary Models2025.08 | 34.62 | 39.5 | 29.5 | 31.5 | 38 | 65.35 | 74.59 | 33.33 | 90.67 | 59.94 | |
| ToolACE-MTBackbone=Llama3.1-8B-Inst, Ablation=without Offline Verification2025.08 | 32.5 | 48 | 25.5 | 25.5 | 31 | 79.71 | 75.52 | 83.33 | 80.65 | 63.01 | |
| Multi-Agent SimulationBackbone=Llama3.1-8B-Inst2025.08 | 31.38 | 46.5 | 19 | 31 | 29 | 80.29 | 78.05 | 72.22 | 90.11 | 64.17 | |
| DeepSeek-V3-0324Model Category=Open-Source Models2025.08 | 29.87 | 41 | 21 | 23 | 34.5 | 88.54 | 77.34 | 83.33 | 76.49 | 64.71 | |
| ToolACE-MTBackbone=Llama3.1-8B-Inst, Ablation=without Iterative Refinement2025.08 | 20.88 | 39 | 12 | 10.5 | 22 | 75.92 | 61.57 | 72.22 | 46.25 | 52.1 | |
| Llama3.1-70B-InstModel Category=Open-Source Models2025.08 | 12.5 | 17 | 13 | 10.5 | 9.5 | 89.98 | 62.24 | 100 | 54.78 | 53.57 | |
| Llama3.1-8B-InstModel Category=Open-Source Models2025.08 | 9.25 | 12 | 10 | 7 | 8 | 84.21 | 61.08 | 77.78 | 48.82 | 49.57 |