Single-hop Tool Calling on WHEN2TOOL single-hop 1.0 (test)
94.3AccuracyPROBE&PREFILL
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PROBE&PREFILLBackbone=Qwen3-14B, tau=0.12026.05 | 94.3 | 2,128 | |
| Prompt-onlyBackbone=Qwen3-32B, Setting=Default2026.05 | 94.1 | 2,404 | |
| PROBE&PREFILLBackbone=Qwen3-14B, tau=0.32026.05 | 94.1 | 1,509 | |
| PROBE&PREFILLBackbone=Qwen3-32B, tau=0.12026.05 | 94 | 1,996 | |
| Prompt-onlyBackbone=Qwen3-14B, Setting=Force2026.05 | 93.7 | 2,421 | |
| Prompt-onlyBackbone=Qwen3-14B, Setting=Default2026.05 | 93.7 | 2,211 | |
| Reason-then-ActBackbone=Qwen3-32B, Setting=Force2026.05 | 93.7 | 2,150 | |
| Prompt-onlyBackbone=Qwen3-14B, Setting=Necessary2026.05 | 93.5 | 2,175 | |
| Prompt-onlyBackbone=Qwen3-32B, Setting=Necessary2026.05 | 93.5 | 2,394 | |
| Reason-then-ActBackbone=Qwen3-14B, Setting=Force2026.05 | 93.4 | 2,172 | |
| Prompt-onlyBackbone=Qwen3-32B, Setting=Sparse2026.05 | 93.3 | 1,912 | |
| Reason-then-ActBackbone=Qwen3-32B, Setting=Necessary2026.05 | 93.2 | 1,845 | |
| PROBE&PREFILLBackbone=Qwen3-32B, tau=0.32026.05 | 93.2 | 1,493 | |
| Reason-then-ActBackbone=Qwen3-32B, Setting=Default2026.05 | 92.9 | 1,823 | |
| Prompt-onlyBackbone=Qwen3-32B, Setting=Force2026.05 | 92.8 | 2,559 | |
| Reason-then-ActBackbone=Qwen3-14B, Setting=Necessary2026.05 | 92.4 | 1,634 | |
| PROBE&PREFILLBackbone=Qwen3-14B, tau=0.52026.05 | 92.4 | 1,227 | |
| Prompt-onlyBackbone=Qwen3-4B, Setting=Force2026.05 | 92 | 2,435 | |
| Reason-then-ActBackbone=Qwen3-14B, Setting=Default2026.05 | 92 | 1,589 | |
| PROBE&PREFILLBackbone=Qwen3-4B, tau=0.12026.05 | 91.1 | 2,216 | |
| Reason-then-ActBackbone=Qwen3-4B, Setting=Force2026.05 | 90.9 | 1,651 | |
| PROBE&PREFILLBackbone=Qwen3-4B, tau=0.32026.05 | 90.4 | 1,707 | |
| PROBE&PREFILLBackbone=Qwen3-32B, tau=0.52026.05 | 90.1 | 1,155 | |
| Prompt-onlyBackbone=Qwen3-4B, Setting=Default2026.05 | 89.2 | 2,118 | |
| PROBE&PREFILLBackbone=Llama-70B, tau=0.92026.05 | 89.2 | 2,804 | |
| PROBE&PREFILLBackbone=Qwen3-1.7B, tau=0.32026.05 | 89 | 2,507 | |
| PROBE&PREFILLBackbone=Qwen3-1.7B, tau=0.12026.05 | 88.8 | 2,512 | |
| PROBE&PREFILLBackbone=Llama-70B, tau=0.32026.05 | 88.6 | 2,902 | |
| PROBE&PREFILLBackbone=Llama-70B, tau=0.72026.05 | 88.6 | 2,828 | |
| PROBE&PREFILLBackbone=Qwen3-4B, tau=0.52026.05 | 88.5 | 1,309 | |
| PROBE&PREFILLBackbone=Llama-70B, tau=0.12026.05 | 88.4 | 2,976 | |
| PROBE&PREFILLBackbone=Qwen3-1.7B, tau=0.52026.05 | 88.3 | 2,128 | |
| PROBE&PREFILLBackbone=Llama-70B, tau=0.52026.05 | 88.3 | 2,871 | |
| Prompt-onlyBackbone=Qwen3-1.7B, Setting=Default2026.05 | 88.2 | 2,709 | |
| Reason-then-ActBackbone=Qwen3-32B, Setting=Sparse2026.05 | 87.9 | 1,231 | |
| Prompt-onlyBackbone=Qwen3-1.7B, Setting=Necessary2026.05 | 87.2 | 2,787 | |
| Prompt-onlyBackbone=Qwen3-1.7B, Setting=Force2026.05 | 86.8 | 3,120 | |
| Reason-then-ActBackbone=Qwen3-1.7B, Setting=Force2026.05 | 86.7 | 2,475 | |
| PROBE&PREFILLBackbone=Qwen3-14B, tau=0.72026.05 | 85.8 | 907 | |
| Prompt-onlyBackbone=Qwen3-4B, Setting=Necessary2026.05 | 85.7 | 1,852 | |
| Reason-then-ActBackbone=Qwen3-14B, Setting=Sparse2026.05 | 85.5 | 1,034 | |
| PROBE&PREFILLBackbone=Qwen3-4B, tau=0.72026.05 | 84.8 | 1,026 | |
| Reason-then-ActBackbone=Qwen3-1.7B, Setting=Default2026.05 | 84.7 | 1,923 | |
| Prompt-onlyBackbone=Llama-70B, Setting=Necessary2026.05 | 84.4 | 3,988 | |
| Prompt-onlyBackbone=Llama-70B, Setting=Sparse2026.05 | 84.2 | 3,431 | |
| Reason-then-ActBackbone=Qwen3-1.7B, Setting=Necessary2026.05 | 84 | 1,871 | |
| Reason-then-ActBackbone=Qwen3-14B, Setting=No-tool2026.05 | 83.5 | 928 | |
| Reason-then-ActBackbone=Qwen3-4B, Setting=Default2026.05 | 83.4 | 1,024 | |
| Reason-then-ActBackbone=Qwen3-4B, Setting=Necessary2026.05 | 83.4 | 1,027 | |
| Prompt-onlyBackbone=Llama-70B, Setting=Default2026.05 | 83.1 | 4,377 | |
| Prompt-onlyBackbone=Llama-70B, Setting=No-tool2026.05 | 82.5 | 3,615 | |
| PROBE&PREFILLBackbone=Qwen3-32B, tau=0.72026.05 | 82.3 | 896 | |
| PROBE&PREFILLBackbone=Qwen3-1.7B, tau=0.72026.05 | 81.6 | 1,415 | |
| Prompt-onlyBackbone=Llama-8B, Setting=Sparse2026.05 | 81.4 | 2,868 | |
| Prompt-onlyBackbone=Llama-8B, Setting=Default2026.05 | 79.5 | 3,708 | |
| Prompt-onlyBackbone=Llama-8B, Setting=Force2026.05 | 78.7 | 4,081 | |
| Prompt-onlyBackbone=Llama-70B, Setting=Force2026.05 | 78.3 | 5,302 | |
| Prompt-onlyBackbone=Llama-8B, Setting=Necessary2026.05 | 77.7 | 3,548 | |
| Prompt-onlyBackbone=Llama-8B, Setting=No-tool2026.05 | 77.6 | 3,391 | |
| Prompt-onlyBackbone=Qwen3-14B, Setting=Sparse2026.05 | 77.3 | 1,020 | |
| PROBE&PREFILLBackbone=Qwen3-4B, tau=0.92026.05 | 74.7 | 657 | |
| Reason-then-ActBackbone=Qwen3-32B, Setting=No-tool2026.05 | 74.3 | 596 | |
| Reason-then-ActBackbone=Qwen3-1.7B, Setting=Sparse2026.05 | 73.2 | 1,161 | |
| Reason-then-ActBackbone=Qwen3-4B, Setting=Sparse2026.05 | 72.5 | 535 | |
| PROBE&PREFILLBackbone=Qwen3-32B, tau=0.92026.05 | 71.5 | 604 | |
| Reason-then-ActBackbone=Qwen3-1.7B, Setting=No-tool2026.05 | 71.1 | 1,117 | |
| Prompt-onlyBackbone=Qwen3-4B, Setting=Sparse2026.05 | 70.7 | 484 | |
| Prompt-onlyBackbone=Qwen3-1.7B, Setting=Sparse2026.05 | 70.2 | 1,611 | |
| PROBE&PREFILLBackbone=Llama-8B, tau=0.52026.05 | 69.7 | 2,381 | |
| PROBE&PREFILLBackbone=Llama-8B, tau=0.12026.05 | 69.2 | 3,027 | |
| PROBE&PREFILLBackbone=Llama-8B, tau=0.32026.05 | 68.9 | 2,770 | |
| Prompt-onlyBackbone=Qwen3-32B, Setting=No-tool2026.05 | 66.8 | 451 | |
| PROBE&PREFILLBackbone=Llama-8B, tau=0.72026.05 | 66.5 | 2,146 | |
| PROBE&PREFILLBackbone=Qwen3-14B, tau=0.92026.05 | 66 | 347 | |
| Reason-then-ActBackbone=Qwen3-4B, Setting=No-tool2026.05 | 64.5 | 315 | |
| PROBE&PREFILLBackbone=Llama-8B, tau=0.92026.05 | 61.7 | 1,753 | |
| Reason-then-ActBackbone=Llama-70B, Setting=No-tool2026.05 | 54.8 | 0 | |
| Reason-then-ActBackbone=Llama-70B, Setting=Sparse2026.05 | 54.1 | 0 | |
| Prompt-onlyBackbone=Qwen3-14B, Setting=No-tool2026.05 | 51.2 | 85 | |
| Prompt-onlyBackbone=Qwen3-4B, Setting=No-tool2026.05 | 50.8 | 121 | |
| Reason-then-ActBackbone=Llama-70B, Setting=Default2026.05 | 47.9 | 0 | |
| PROBE&PREFILLBackbone=Qwen3-1.7B, tau=0.92026.05 | 47.9 | 293 | |
| Reason-then-ActBackbone=Llama-70B, Setting=Necessary2026.05 | 47.6 | 0 | |
| Reason-then-ActBackbone=Llama-70B, Setting=Force2026.05 | 42.4 | 0 | |
| Reason-then-ActBackbone=Llama-8B, Setting=Sparse2026.05 | 36.3 | 0 | |
| Reason-then-ActBackbone=Llama-8B, Setting=No-tool2026.05 | 34.5 | 6 | |
| Reason-then-ActBackbone=Llama-8B, Setting=Necessary2026.05 | 32 | 11 | |
| Reason-then-ActBackbone=Llama-8B, Setting=Default2026.05 | 31.2 | 2 | |
| Reason-then-ActBackbone=Llama-8B, Setting=Force2026.05 | 29.3 | 9 | |
| Prompt-onlyBackbone=Qwen3-1.7B, Setting=No-tool2026.05 | 26.3 | 293 |