Tool Use on BFCL V4
76.5AccuracyClaude-Opus-4.5
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Claude-Opus-4.52026.06 | 76.5 | — | — | — | |
| Seed2.0 Pro2026.06 | 73.4 | — | — | — | |
| Claude-Sonnet-4.52026.06 | 72.9 | — | — | — | |
| Seed2.0 Lite2026.06 | 72.9 | — | — | — | |
| Gemini-3-pro High2026.06 | 71 | — | — | — | |
| GPT-5.2 High2026.06 | 65.9 | — | — | — | |
| Gemini-3-Flash High2026.06 | 65 | — | — | — | |
| GPT-5-mini High2026.06 | 57.9 | — | — | — | |
| Nanbeige4.1-3BParameters=3B2026.02 | 56.5 | — | — | — | |
| Nanbeige4-3B-2511Parameters=3B2026.02 | 53.8 | — | — | — | |
| Qwen3-Next-80B-A3BParameters=80B2026.02 | 50.51 | — | — | — | |
| Qwen3-30B-A3B-2507Parameters=30B2026.02 | 48.6 | — | — | — | |
| Qwen3-32BParameters=32B2026.02 | 47.9 | — | — | — | |
| Mellum 2 (RL)Post-training Stage=RL, Parameters=2.5B/12B, Variant=Thinking2026.05 | 45.6 | — | — | — | |
| Qwen3-4B-2507Parameters=4B2026.02 | 44.87 | — | — | — | |
| Qwen3.5-4BParameters=4B, Variant=Thinking2026.05 | 42.9 | — | — | — | |
| Qwen3.5-9BParameters=9B, Variant=Thinking2026.05 | 42.7 | — | — | — | |
| Mellum 2 (SFT)Post-training Stage=SFT, Parameters=2.5B/12B, Variant=Thinking2026.05 | 38.8 | — | — | — | |
| GDPO+SAWBackbone=Qwen2.5-3B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 36.62 | 35.98 | 56.4 | 4.12 | |
| GDPO+GradientBackbone=Qwen2.5-3B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 36.43 | 36.31 | 55.22 | 4.88 | |
| GRPO+SAWBackbone=Qwen2.5-3B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 36.35 | 36.06 | 55.22 | 4.88 | |
| GRPOBackbone=Qwen2.5-3B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 36.27 | 35.94 | 55.29 | 4.63 | |
| Ministral-3-14BParameters=14B, Variant=Thinking2026.05 | 35.9 | — | — | — | |
| GRPO+GradientBackbone=Qwen2.5-3B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 35.6 | 35.75 | 54.56 | 3.38 | |
| GDPOBackbone=Qwen2.5-3B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 35.58 | 35.71 | 54.03 | 4.25 | |
| BaseBackbone=Qwen2.5-3B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 34.57 | 35.13 | 52.19 | 4 | |
| GDPO+SAWBackbone=Qwen2.5-1.5B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 34.02 | 33.98 | 52.63 | 2.63 | |
| GDPO+GradientBackbone=Qwen2.5-1.5B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 33.93 | 33.48 | 52.78 | 2.75 | |
| GDPOBackbone=Qwen2.5-1.5B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 33.82 | 34.29 | 51.96 | 2.5 | |
| GRPO+SAWBackbone=Qwen2.5-1.5B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 33.53 | 33.1 | 52.19 | 2.63 | |
| GRPO+GradientBackbone=Qwen2.5-1.5B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 33.41 | 33.98 | 51.22 | 2.5 | |
| GRPOBackbone=Qwen2.5-1.5B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 32.55 | 32.81 | 50.26 | 2.25 | |
| BaseBackbone=Qwen2.5-1.5B-Instruct, Evaluation Protocol=averaged over 4 evaluation runs2026.06 | 31.25 | 30.92 | 48.78 | 2.13 |