Tool Calling on DIABENCH Static Evaluation 1.0
82AccuracyLlama-3.3-Nemotron-DiaFORGE-49B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Llama-3.3-Nemotron-DiaFORGE-49BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 82 | 4 | 12 | |
| Gemma-3-DiaFORGE-27BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 77 | 3 | 18 | |
| Gemma-3-DiaFORGE-12BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 68 | 2 | 26 | |
| GPT-4o-20241120-fcNative tool calling support=true, Prompting=Native function/tool calling2025.07 | 61 | 64 | 16 | |
| Llama-3.3-Nemotron-Super-49BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 60 | 7 | 25 | |
| Gemma-3-DiaFORGE-4BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 53 | 5 | 37 | |
| Llama-3.2-DiaFORGE-3BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 52 | 12 | 30 | |
| Llama-xLAM-2-70B-fc-rNative tool calling support=true, Prompting=Native function/tool calling, Architecture=instruction-tuned, decoder-only2025.07 | 48 | 18 | 13 | |
| Claude-3.5-Sonnet-20241022-fcNative tool calling support=true, Prompting=Native function/tool calling2025.07 | 42 | 76 | 3 | |
| Llama-3.3-DiaFORGE-70BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 38 | 3 | 58 | |
| Gemma-3-12BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 31 | 3 | 62 | |
| Llama-3.3-70B-fcNative tool calling support=true, Prompting=Native function/tool calling, Architecture=instruction-tuned, decoder-only2025.07 | 22 | 52 | 1 | |
| Gemma-3-4BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 19 | 17 | 61 | |
| Gemma-3-27BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 19 | 2 | 78 | |
| GPT-4o-20241120Native tool calling support=false, Prompting=CAPO-optimized system prompts2025.07 | 19 | 0 | 81 | |
| Claude-3.5-Sonnet-20241022Native tool calling support=false, Prompting=CAPO-optimized system prompts2025.07 | 15 | 2 | 82 | |
| Llama-3.3-70BNative tool calling support=false, Prompting=CAPO-optimized system prompts, Architecture=instruction-tuned, decoder-only2025.07 | 3 | 0 | 97 |