Agent Task on Sudoku
99Success Rate (SR)Gemini 2.5 Pro
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Gemini 2.5 ProModel Category=Thinking Model2026.02 | 99 | 60.2 | |
| Gemini 2.5 FlashModel Category=Non-thinking Model2026.02 | 97 | 60.4 | |
| gpt-oss-120bModel Category=Thinking Model2026.02 | 97 | 60.2 | |
| Qwen3-30B-A3B-ThinkingModel Category=Thinking Model2026.02 | 95 | 55.6 | |
| DeepSeek-R1Model Category=Thinking Model2026.02 | 95 | 59.4 | |
| DeepSeek-V3.2Model Category=Non-thinking Model2026.02 | 93 | 56.8 | |
| Qwen3-30B-A3B-InstructModel Category=Non-thinking Model2026.02 | 77 | 48.8 | |
| GLM-4-32B-0414Model Category=Non-thinking Model2026.02 | 55 | 34.4 | |
| Qwen3-4B-ThinkingModel Category=Thinking Model2026.02 | 55 | 30.7 | |
| Phi-4-reasoningModel Category=Thinking Model2026.02 | 51 | 37.4 | |
| Llama-3.3-70B-InstructModel Category=Non-thinking Model2026.02 | 40 | 24.9 | |
| Phi-4Model Category=Non-thinking Model2026.02 | 33 | 21.4 | |
| Qwen3-4B-InstructModel Category=Non-thinking Model2026.02 | 29 | 19.3 | |
| Ministral-3-14B-InstructModel Category=Non-thinking Model2026.02 | 18 | 13.3 | |
| Llama-3.1-8B-InstructModel Category=Non-thinking Model2026.02 | 4 | 2.1 | |
| GLM-4-9B-ChatModel Category=Non-thinking Model2026.02 | 4 | 2 | |
| Mistral-7B-InstructModel Category=Non-thinking Model2026.02 | 1 | 0.4 |