Reasoning on Checkmate-in-One
92AccuracyRoT
Evaluation Results
| Method | Links | |
|---|---|---|
| RoTBackbone=GPT-4, Shots=2-shot2024.10 | 92 | |
| RoTBackbone=GPT-4, Shots=1-shot2024.10 | 91.2 | |
| BOTBackbone=GPT-4, Shots=2-shot2024.10 | 88.3 | |
| BOTBackbone=GPT-4, Shots=1-shot2024.10 | 87.1 | |
| RoTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 72.6 | |
| RoTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 71.5 | |
| BOTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 68.5 | |
| BOTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 67.9 | |
| Meta-promptingBackbone=GPT-4, Shots=2-shot2024.10 | 58.9 | |
| Meta-promptingBackbone=GPT-4, Shots=1-shot2024.10 | 58.2 | |
| GOTBackbone=GPT-4, Shots=2-shot2024.10 | 51.2 | |
| TOTBackbone=GPT-4, Shots=1-shot2024.10 | 49.8 | |
| GOTBackbone=GPT-4, Shots=1-shot2024.10 | 49 | |
| TOTBackbone=GPT-4, Shots=2-shot2024.10 | 48.8 | |
| Meta-promptingBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 48 | |
| Meta-promptingBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 47.4 | |
| GOTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 46 | |
| GOTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 45.4 | |
| TOTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 45.2 | |
| TOTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 44.2 | |
| COTBackbone=GPT-4, Shots=2-shot2024.10 | 36.7 | |
| COTBackbone=GPT-4, Shots=1-shot2024.10 | 35.8 | |
| COTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 25.7 | |
| COTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 25.4 | |
| MetaTuner-JBase Model=Llama3.2-3B2025.09 | 21.43 | |
| MetaTuner-JBase Model=Qwen2.5-7B2025.09 | 20.86 | |
| BetterTogetherBase Model=Llama3.2-3B2025.09 | 20.57 | |
| KTOBase Model=Llama3.2-3B2025.09 | 20 | |
| BetterTogetherBase Model=Qwen2.5-7B2025.09 | 19.71 | |
| PPOBase Model=Llama3.2-3B2025.09 | 19.43 | |
| DPOBase Model=Llama3.2-3B2025.09 | 19.14 | |
| DPOBase Model=Qwen2.5-7B2025.09 | 18.86 | |
| SFTBase Model=Llama3.2-3B2025.09 | 18.85 | |
| KTOBase Model=Qwen2.5-7B2025.09 | 18.57 | |
| PPOBase Model=Qwen2.5-7B2025.09 | 18.29 | |
| SFTBase Model=Qwen2.5-7B2025.09 | 18 | |
| MetaTuner-JBase Model=Qwen2.5-3B2025.09 | 17.71 | |
| BetterTogetherBase Model=Qwen2.5-3B2025.09 | 15.14 | |
| KTOBase Model=Qwen2.5-3B2025.09 | 14.57 | |
| PPOBase Model=Qwen2.5-3B2025.09 | 14 | |
| SFTBase Model=Qwen2.5-3B2025.09 | 13.71 | |
| DPOBase Model=Qwen2.5-3B2025.09 | 13.42 | |
| BPOBase Model=Llama3.2-3B2025.09 | 1.43 | |
| CFPOBase Model=Llama3.2-3B2025.09 | 1.43 | |
| OPROBase Model=Llama3.2-3B2025.09 | 1.14 | |
| BPOBase Model=Qwen2.5-7B2025.09 | 0.86 | |
| OPROBase Model=Qwen2.5-7B2025.09 | 0.57 | |
| Base ModelBase Model=Qwen2.5-3B2025.09 | 0 | |
| Base ModelBase Model=Qwen2.5-7B2025.09 | 0 | |
| Base ModelBase Model=Llama3.2-3B2025.09 | 0 | |
| RLPromptBase Model=Qwen2.5-3B2025.09 | 0 | |
| RLPromptBase Model=Qwen2.5-7B2025.09 | 0 | |
| RLPromptBase Model=Llama3.2-3B2025.09 | 0 | |
| BPOBase Model=Qwen2.5-3B2025.09 | 0 | |
| OPROBase Model=Qwen2.5-3B2025.09 | 0 | |
| CFPOBase Model=Qwen2.5-3B2025.09 | 0 | |
| CFPOBase Model=Qwen2.5-7B2025.09 | 0 |