List your company
ResearchFormerly WizWand

Symbolic Reasoning on Last Letter Concatenation

90.4AccuracyZero-Shot CoT
-3.61620.79245.269.608Oct 7, 2022May 14, 2023Dec 19, 2023Jul 25, 2024Mar 1, 2025Oct 6, 2025May 13, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Zero-Shot CoTPrompting Technique=Zero-Shot Chain-of-Thought2023.1090.4
Persona SwitchModel=LLaMA-3.1 (8B), Zero-shot=true2026.0184.2
GreedyModel=LLaMA-3.1 (8B), Zero-shot=true2026.0182.8
RM-RegenModel=Llama3.1-8B2026.0379.33
MultinomialModel=LLaMA-3.1 (8B), Zero-shot=true2026.0178
Random SelectionModel=LLaMA-3.1 (8B), Zero-shot=true2026.0177.47
RM-RegenModel=GPT-3.52026.0377.33
Low-gap SelectionModel=LLaMA-3.1 (8B), Zero-shot=true2026.0175.4
Manual CoT + SCBase Model=gpt-3.5-turbo2023.0674.5
Role-Play PromptingModel=LLaMA-3.1 (8B), Zero-shot=true2026.0174.2
ProCoModel=GPT-3.5, iterations=32026.0374
Manual CoTBase Model=gpt-3.5-turbo2023.0673
Persona SwitchModel=LLaMA-3.2 (3B), Zero-shot=true2026.0172.8
ST CoTModel=GPT-3.5, iterations=32026.0372.67
Top-pModel=LLaMA-3.1 (8B), Zero-shot=true2026.0172.2
ProCoModel=Llama3.1-8B, iterations=32026.0371.33
ZS-CoT (teacher)Model=GPT-4o-mini, #Params=-2026.0571.2
CoK + SC + F2-VBase Model=gpt-3.5-turbo2023.0669.7
Vanilla-KDModel=T5-large, #Params=780M2026.0569
CoK + SCBase Model=gpt-3.5-turbo2023.0668.3
MultinomialModel=LLaMA-3.2 (3B), Zero-shot=true2026.0166.8
Top-kModel=LLaMA-3.1 (8B), Zero-shot=true2026.0166.8
GateKDModel=T5-base, #Params=250M2026.0565.7
GreedyModel=LLaMA-3.2 (3B), Zero-shot=true2026.0165.4
CoK + F2-VBase Model=gpt-3.5-turbo2023.0665.4
CoKBase Model=gpt-3.5-turbo2023.0663.1
Random SelectionModel=LLaMA-3.2 (3B), Zero-shot=true2026.0162.53
CoK + F2-VBase Model=text-davinci-0022023.0661.1
Top-pModel=LLaMA-3.2 (3B), Zero-shot=true2026.0161
Mentor-KDModel=T5-base, #Params=250M2026.0561
GateKDModel=T5-small, #Params=80M2026.0560.1
Auto-CoTPrompting=Automatic Chain-of-Thought2022.1059.7
Auto-CoTBase Model=text-davinci-0022023.0659.7
CoKBase Model=text-davinci-0022023.0659.4
Manual-CoTPrompting=Manual Chain-of-Thought2022.1059
Manual CoTBase Model=text-davinci-0022023.0659
PROMPTEDPrompting Technique=PROMPTED2023.1058.2
MCC-KDModel=T5-base, #Params=250M2026.0558.2
ST CoTModel=Llama3.1-8B, iterations=32026.0358
Zero-Shot-CoTPrompting=Zero-shot Chain-of-Thought2022.1057.6
Zero-Shot CoTBase Model=text-davinci-0022023.0657.6
Vanilla-KDModel=T5-base, #Params=250M2026.0556.1
Role-Play PromptingModel=LLaMA-3.2 (3B), Zero-shot=true2026.0156
Low-gap SelectionModel=LLaMA-3.2 (3B), Zero-shot=true2026.0155.2
Mentor-KDModel=T5-small, #Params=80M2026.0555.2
Top-kModel=LLaMA-3.2 (3B), Zero-shot=true2026.0154.8
MCC-KDModel=T5-small, #Params=80M2026.0551.3
Output RefinementPrompting Technique=Output Refinement2023.1050.8
Vanilla-KDModel=T5-small, #Params=80M2026.0549.6
Self-RefineModel=GPT-3.5, iterations=32026.0346
Self-RefineModel=Llama3.1-8B, iterations=32026.0340
RM-RegenModel=Gemma2-9B2026.0337.86
ProCoModel=Gemma2-9B, iterations=32026.0335.33
ST CoTModel=Gemma2-9B, iterations=32026.0333.33
Self-RefineModel=Gemma2-9B, iterations=32026.0324
Persona SwitchModel=Gemma-2 (2B), Zero-shot=true2026.0113.6
GreedyModel=Gemma-2 (2B), Zero-shot=true2026.0111
Random SelectionModel=Gemma-2 (2B), Zero-shot=true2026.017.86
Role-Play PromptingModel=Gemma-2 (2B), Zero-shot=true2026.016.2
Top-pModel=Gemma-2 (2B), Zero-shot=true2026.016
Low-gap SelectionModel=Gemma-2 (2B), Zero-shot=true2026.015.4
MultinomialModel=Gemma-2 (2B), Zero-shot=true2026.015.2
Top-kModel=Gemma-2 (2B), Zero-shot=true2026.014.8
Zero-ShotPrompting Technique=Zero-Shot2023.103.2
Zero-ShotPrompting=Zero-shot2022.100.2
Few-ShotPrompting=Few-shot2022.100.2
Zero-Shot SPBase Model=text-davinci-0022023.060.2
Few-Shot SPBase Model=text-davinci-0022023.060