Symbolic Reasoning on Last Letter Concatenation
90.4AccuracyZero-Shot CoT
Evaluation Results
| Method | Links | |
|---|---|---|
| Zero-Shot CoTPrompting Technique=Zero-Shot Chain-of-Thought2023.10 | 90.4 | |
| Persona SwitchModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 84.2 | |
| GreedyModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 82.8 | |
| RM-RegenModel=Llama3.1-8B2026.03 | 79.33 | |
| MultinomialModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 78 | |
| Random SelectionModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 77.47 | |
| RM-RegenModel=GPT-3.52026.03 | 77.33 | |
| Low-gap SelectionModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 75.4 | |
| Manual CoT + SCBase Model=gpt-3.5-turbo2023.06 | 74.5 | |
| Role-Play PromptingModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 74.2 | |
| ProCoModel=GPT-3.5, iterations=32026.03 | 74 | |
| Manual CoTBase Model=gpt-3.5-turbo2023.06 | 73 | |
| Persona SwitchModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 72.8 | |
| ST CoTModel=GPT-3.5, iterations=32026.03 | 72.67 | |
| Top-pModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 72.2 | |
| ProCoModel=Llama3.1-8B, iterations=32026.03 | 71.33 | |
| ZS-CoT (teacher)Model=GPT-4o-mini, #Params=-2026.05 | 71.2 | |
| CoK + SC + F2-VBase Model=gpt-3.5-turbo2023.06 | 69.7 | |
| Vanilla-KDModel=T5-large, #Params=780M2026.05 | 69 | |
| CoK + SCBase Model=gpt-3.5-turbo2023.06 | 68.3 | |
| MultinomialModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 66.8 | |
| Top-kModel=LLaMA-3.1 (8B), Zero-shot=true2026.01 | 66.8 | |
| GateKDModel=T5-base, #Params=250M2026.05 | 65.7 | |
| GreedyModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 65.4 | |
| CoK + F2-VBase Model=gpt-3.5-turbo2023.06 | 65.4 | |
| CoKBase Model=gpt-3.5-turbo2023.06 | 63.1 | |
| Random SelectionModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 62.53 | |
| CoK + F2-VBase Model=text-davinci-0022023.06 | 61.1 | |
| Top-pModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 61 | |
| Mentor-KDModel=T5-base, #Params=250M2026.05 | 61 | |
| GateKDModel=T5-small, #Params=80M2026.05 | 60.1 | |
| Auto-CoTPrompting=Automatic Chain-of-Thought2022.10 | 59.7 | |
| Auto-CoTBase Model=text-davinci-0022023.06 | 59.7 | |
| CoKBase Model=text-davinci-0022023.06 | 59.4 | |
| Manual-CoTPrompting=Manual Chain-of-Thought2022.10 | 59 | |
| Manual CoTBase Model=text-davinci-0022023.06 | 59 | |
| PROMPTEDPrompting Technique=PROMPTED2023.10 | 58.2 | |
| MCC-KDModel=T5-base, #Params=250M2026.05 | 58.2 | |
| ST CoTModel=Llama3.1-8B, iterations=32026.03 | 58 | |
| Zero-Shot-CoTPrompting=Zero-shot Chain-of-Thought2022.10 | 57.6 | |
| Zero-Shot CoTBase Model=text-davinci-0022023.06 | 57.6 | |
| Vanilla-KDModel=T5-base, #Params=250M2026.05 | 56.1 | |
| Role-Play PromptingModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 56 | |
| Low-gap SelectionModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 55.2 | |
| Mentor-KDModel=T5-small, #Params=80M2026.05 | 55.2 | |
| Top-kModel=LLaMA-3.2 (3B), Zero-shot=true2026.01 | 54.8 | |
| MCC-KDModel=T5-small, #Params=80M2026.05 | 51.3 | |
| Output RefinementPrompting Technique=Output Refinement2023.10 | 50.8 | |
| Vanilla-KDModel=T5-small, #Params=80M2026.05 | 49.6 | |
| Self-RefineModel=GPT-3.5, iterations=32026.03 | 46 | |
| Self-RefineModel=Llama3.1-8B, iterations=32026.03 | 40 | |
| RM-RegenModel=Gemma2-9B2026.03 | 37.86 | |
| ProCoModel=Gemma2-9B, iterations=32026.03 | 35.33 | |
| ST CoTModel=Gemma2-9B, iterations=32026.03 | 33.33 | |
| Self-RefineModel=Gemma2-9B, iterations=32026.03 | 24 | |
| Persona SwitchModel=Gemma-2 (2B), Zero-shot=true2026.01 | 13.6 | |
| GreedyModel=Gemma-2 (2B), Zero-shot=true2026.01 | 11 | |
| Random SelectionModel=Gemma-2 (2B), Zero-shot=true2026.01 | 7.86 | |
| Role-Play PromptingModel=Gemma-2 (2B), Zero-shot=true2026.01 | 6.2 | |
| Top-pModel=Gemma-2 (2B), Zero-shot=true2026.01 | 6 | |
| Low-gap SelectionModel=Gemma-2 (2B), Zero-shot=true2026.01 | 5.4 | |
| MultinomialModel=Gemma-2 (2B), Zero-shot=true2026.01 | 5.2 | |
| Top-kModel=Gemma-2 (2B), Zero-shot=true2026.01 | 4.8 | |
| Zero-ShotPrompting Technique=Zero-Shot2023.10 | 3.2 | |
| Zero-ShotPrompting=Zero-shot2022.10 | 0.2 | |
| Few-ShotPrompting=Few-shot2022.10 | 0.2 | |
| Zero-Shot SPBase Model=text-davinci-0022023.06 | 0.2 | |
| Few-Shot SPBase Model=text-davinci-0022023.06 | 0 |