Grounded Chess Reasoning on Chess Puzzles (test)
95Accuracy (Beginner)gpt-5
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| gpt-5Model Category=Proprietary models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 95 | 84 | 54 | 31 | 85.2 | 76.7 | 12,193 | |
| gemini-3-proModel Category=Proprietary models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 88 | 86 | 70 | 44 | 78.2 | 75.4 | 3,182 | |
| gemini-3-flashModel Category=Proprietary models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 65 | 59 | 34 | 19 | 38 | 40.8 | 6,418 | |
| C1-4BModel Category=Ours, Evaluation Protocol=Pass@12026.03 | 65 | 39 | 39 | 22 | 53.6 | 48.1 | 178 | |
| C1-SFT-8BModel Category=Ours, Fine-tuning Strategy=SFT, Evaluation Protocol=Pass@12026.03 | 57 | 36 | 27 | 27 | 46.6 | 42.2 | 189 | |
| gpt-5-chatModel Category=Proprietary models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 52 | 39 | 27 | 18 | 41.8 | 38.3 | 925 | |
| C1-SFT-4BModel Category=Ours, Fine-tuning Strategy=SFT, Evaluation Protocol=Pass@12026.03 | 51 | 30 | 30 | 26 | 46.2 | 40.9 | 188 | |
| gemini-2.5-proModel Category=Proprietary models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 37 | 31 | 29 | 19 | 31 | 30.1 | 9,668 | |
| claude-sonnet-4Model Category=Proprietary models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 35 | 19 | 16 | 10 | 26.8 | 23.8 | 8,028 | |
| claude-haiku-4.5Model Category=Proprietary models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 33 | 24 | 14 | 11 | 25.6 | 23.3 | 8,111 | |
| claude-sonnet-4.5Model Category=Proprietary models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 32 | 29 | 15 | 11 | 28.6 | 25.6 | 3,227 | |
| deepseek-chat-v3.1Model Category=Open-source models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 27 | 21 | 6 | 16 | 22 | 20 | 11,249 | |
| qwen3-next-80b-a3bModel Category=Open-source models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 24 | 14 | 14 | 8 | 17.6 | 16.4 | 13,938 | |
| qwen3-maxModel Category=Open-source models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 22 | 15 | 3 | 16 | 13.8 | 13.9 | 3,393 | |
| llama-4-maverickModel Category=Open-source models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 12 | 8 | 5 | 10 | 8.6 | 8.7 | 1,092 | |
| deepseek-r1-0528Model Category=Open-source models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 11 | 10 | 14 | 16 | 16 | 14.6 | 14,442 | |
| gemini-2.5-flashModel Category=Proprietary models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 9 | 4 | 6 | 5 | 8.2 | 7.2 | 9,991 | |
| mistral-medium-3.1Model Category=Open-source models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 9 | 6 | 7 | 4 | 8 | 7.3 | 2,818 | |
| llama-4-scoutModel Category=Open-source models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 0 | 0 | 0 | 1 | 0.4 | 0.3 | 806 | |
| gemma-3-27bModel Category=Open-source models, Reasoning Mode=Enabled where applicable, Evaluation Protocol=Pass@12026.03 | 0 | 0 | 0 | 0 | 0 | 0 | 705 |