Function Calling on BFCL V4 (Custom Metrics Set)
62.3Multi-Turn Success RateSeed 2.0
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Seed 2.0Model Category=Frontier Proprietary Models2026.04 | 62.3 | 92 | 57.8 | 89 | 82.2 | 76.6 | 75 | 73.4 | |
| Claude Sonnet-4.5Model Category=Frontier Proprietary Models2026.04 | 61.4 | 81 | 65 | 88.7 | 81.1 | 68.8 | 86.6 | 73.2 | |
| Gemini-3 ProModel Category=Frontier Proprietary Models2026.04 | 60.8 | 80 | 61.7 | 90.7 | 83.1 | 68.8 | 85.6 | 72.5 | |
| Agent-World-14BModel Category=Open-Source Environment Scaling Methods, Parameter Scale=14B2026.04 | 53.9 | 53 | 23.9 | 82.3 | 79.3 | 93.8 | 81 | 55.8 | |
| 4B-ThinkingCategory=Open-source Baselines2026.06 | 52 | 6 | 12.37 | 38.02 | 71.06 | — | 79.23 | 38.61 | |
| 235B-A22B-InstructCategory=Open-source Baselines2026.06 | 50.12 | 44.5 | 27.1 | 90.1 | 83.2 | — | 82.38 | 53.54 | |
| Turn-level TRUSTBase Model=Qwen3-4B-Thinking, Training Level=Turn-level2026.06 | 49.62 | 3 | 24.95 | 89.23 | 84.09 | — | 84.84 | 48.04 | |
| GPT-5.2 HighModel Category=Frontier Proprietary Models2026.04 | 48.5 | 75.5 | 45.8 | 81.9 | 70.4 | 75 | 88.7 | 62.9 | |
| EnvScaler-8BModel Category=Open-Source Environment Scaling Methods, Parameter Scale=8B2026.04 | 47.1 | 23 | 21.9 | 88.5 | 82.2 | 93.8 | 74.6 | 47.6 | |
| Qwen3-235B-A22BModel Category=Open-Source Foundation Models, Parameter Scale=235B2026.04 | 45.4 | 54 | 23.9 | 37.4 | 68.9 | 87.5 | 81.7 | 47.9 | |
| MAESTRO2026.05 | 44.62 | — | — | 87.33 | 82.38 | — | — | 78.09 | |
| Agent-World-8BModel Category=Open-Source Environment Scaling Methods, Parameter Scale=8B2026.04 | 44.5 | 47 | 21.7 | 83.3 | 79.6 | 93.8 | 80.2 | 51.4 | |
| GPT-5.22026.05 | 43.75 | — | — | 78.29 | 67.14 | — | — | 68.58 | |
| Qwen3-32BModel Category=Open-Source Foundation Models, Parameter Scale=32B2026.04 | 43.3 | 26 | 15.7 | 90.3 | 82 | 81.3 | 82.4 | 46.7 | |
| SAGEBase Model=Qwen3-4B-Thinking2026.06 | 41.5 | 3.5 | 13.02 | 86.71 | 80.49 | — | 79.23 | 41.16 | |
| 8B-ThinkingCategory=Open-source Baselines2026.06 | 37.75 | 12 | 14.62 | 87.58 | 80.53 | — | 79.07 | 41.58 | |
| AWM-14BModel Category=Open-Source Environment Scaling Methods, Parameter Scale=14B2026.04 | 37.6 | 10 | 19.8 | 90.2 | 81.5 | 75 | 79.4 | 42.4 | |
| DeepSeek-V3.2-685BModel Category=Open-Source Foundation Models, Parameter Scale=685B2026.04 | 37.4 | 69.5 | 54.2 | 34.9 | 53.7 | 37.5 | 93.2 | 54.1 | |
| Qwen3-14BModel Category=Open-Source Foundation Models, Parameter Scale=14B2026.04 | 36.9 | 4 | 19.8 | 90 | 82.4 | 81.3 | 79.4 | 41 | |
| SAGEBase Model=Qwen3-8B-Thinking2026.06 | 36.37 | 9.5 | 21.72 | 88.15 | 80.83 | — | 79.6 | 42.99 | |
| Gemini-2.5-Flash2026.05 | 36.25 | — | — | 84.96 | 74.39 | — | — | 72.88 | |
| Qwen3-8BModel Category=Open-Source Foundation Models, Parameter Scale=8B2026.04 | 35.4 | 7 | 17.6 | 90.2 | 80.9 | 81.3 | 77.2 | 40.4 | |
| AWM-8BModel Category=Open-Source Environment Scaling Methods, Parameter Scale=8B2026.04 | 34.9 | 9.5 | 15.7 | 90.2 | 80.5 | 93.8 | 73.9 | 40 | |
| 30B-A3B-InstructCategory=Open-source Baselines2026.06 | 30 | 22.5 | 17.63 | 85.77 | 77.94 | — | 79.9 | 41 | |
| Traj.-level TRUSTBase Model=Qwen3-8B-Base, Training Level=Traj.-level2026.06 | 29.88 | 27 | 30.57 | 82.27 | 75.57 | — | 79.33 | 44.48 | |
| AUQBase Model=Qwen3-4B-Thinking2026.06 | 23.5 | 3 | 20.43 | 77.08 | 62.4 | — | 81.37 | 35.21 | |
| Turn-level GRPOBase Model=Qwen3-4B-Thinking, Training Level=Turn-level2026.06 | 22.75 | 3 | 24.52 | 80.61 | 86.33 | — | 82.38 | 38.98 | |
| Traj.-level CM2Base Model=Qwen3-8B-Base, Training Level=Traj.-level2026.06 | 22.5 | 22 | 21.94 | 80.12 | 74.58 | — | 71.46 | 38.15 | |
| TOUCAN-7BModel Category=Open-Source Environment Scaling Methods, Parameter Scale=7B2026.04 | 17.8 | 21 | 18.5 | 81 | 73.9 | 81.3 | 78.6 | 36.6 | |
| Claude-Opus-4.52026.05 | 16.12 | — | — | 89.65 | 76.02 | — | — | 72.14 | |
| AUQBase Model=Qwen3-8B-Thinking2026.06 | 10 | 14 | 15.7 | 84.46 | 75.42 | — | 40.14 | 29.08 | |
| Simulator-8BModel Category=Open-Source Environment Scaling Methods, Parameter Scale=8B2026.04 | 4.1 | 17.5 | 6 | 47.6 | 44.6 | 31.3 | 87.3 | 23.9 |