Tool Calling on DIABENCH Dynamic Evaluation 1.0
89ACCLlama-3.3-Nemotron-DiaFORGE-49B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Llama-3.3-Nemotron-DiaFORGE-49BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 89 | 6 | 3 | |
| Gemma-3-DiaFORGE-27BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 89 | 3 | 3 | |
| Gemma-3-DiaFORGE-12BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 86 | 7 | 7 | |
| Gemma-3-DiaFORGE-4BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 81 | 9 | 5 | |
| Llama-3.2-DiaFORGE-3BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 80 | 8 | 6 | |
| Llama-3.3-DiaFORGE-70BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 79 | 3 | 15 | |
| Llama-3.3-Nemotron-Super-49BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 72 | 8 | 8 | |
| GPT-4o-20241120Native tool calling support=false, Prompting=CAPO-optimized system prompts2025.07 | 62 | 2 | 36 | |
| GPT-4o-20241120-fcNative tool calling support=true, Prompting=Native function/tool calling2025.07 | 56 | 59 | 5 | |
| Llama-xLAM-2-70B-fc-rNative tool calling support=true, Prompting=Native function/tool calling, Architecture=instruction-tuned, decoder-only2025.07 | 51 | 18 | 5 | |
| Claude-3.5-Sonnet-20241022-fcNative tool calling support=true, Prompting=Native function/tool calling2025.07 | 40 | 34 | 3 | |
| Claude-3.5-Sonnet-20241022Native tool calling support=false, Prompting=CAPO-optimized system prompts2025.07 | 39 | 3 | 55 | |
| Gemma-3-12BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 37 | 4 | 57 | |
| Llama-3.3-70B-fcNative tool calling support=true, Prompting=Native function/tool calling, Architecture=instruction-tuned, decoder-only2025.07 | 30 | 22 | 1 | |
| Gemma-3-4BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 24 | 14 | 58 | |
| Gemma-3-27BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 21 | 0 | 79 | |
| Llama-3.3-70BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 11 | 2 | 88 |