Arithmetic Reasoning on SingleEq
98.8AccuracyPAL
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| PALBackbone=GPT-4, Decoding strategy=Greedy2023.05 | 98.8 | — | — | — | |
| Automatic Model Selection with LLMsBackbone=GPT-4, Decoding strategy=Greedy2023.05 | 98.6 | — | — | — | |
| Automatic Model Selection with LLMsBackbone=ChatGPT, Decoding strategy=Greedy2023.05 | 97.8 | — | — | — | |
| PALBackbone=ChatGPT, Decoding strategy=Greedy2023.05 | 97.6 | — | — | — | |
| CoTBackbone=ChatGPT, Decoding strategy=Greedy2023.05 | 97.4 | — | — | — | |
| CoTBackbone=GPT-4, Decoding strategy=Greedy2023.05 | 97.2 | — | — | — | |
| Automatic Model Selection with LLMsBackbone=Codex, Decoding strategy=Greedy2023.05 | 96.9 | — | — | — | |
| PALBackbone=Codex, Decoding strategy=Greedy2023.05 | 96.5 | — | — | — | |
| DUPModel=GPT-4, Evaluation Protocol=Zero-shot2024.04 | 96 | — | — | — | |
| StandPerturbation=No Dert.2025.06 | 95.7 | — | — | — | |
| CoTBackbone=Codex, Decoding strategy=Greedy2023.05 | 95.5 | — | — | — | |
| Zero-shot PS+Model=GPT-4, Evaluation Protocol=Zero-shot2024.04 | 95.3 | — | — | — | |
| RoPPerturbation=EC2025.06 | 95.3 | — | — | — | |
| RoPPerturbation=SC2025.06 | 95.1 | — | — | — | |
| Zero-shot CoTModel=GPT-4, Evaluation Protocol=Zero-shot2024.04 | 95 | — | — | — | |
| Least-to-MostModel=GPT-4, Evaluation Protocol=Zero-shot2024.04 | 95 | — | — | — | |
| DIVERSEModel=code-davinci-0022022.06 | 94.9 | — | — | — | |
| DUPModel=GPT-3.5-Turbo, Evaluation Protocol=Zero-shot2024.04 | 94.9 | — | — | — | |
| Auto-CoTModel=GPT-3.5-Turbo, Evaluation Protocol=Few-shot2024.04 | 94.6 | — | — | — | |
| Manual-CoTModel=GPT-3.5-Turbo, Evaluation Protocol=Few-shot2024.04 | 94.2 | — | — | — | |
| Self-ConsistencyModel=code-davinci-0022022.06 | 93.7 | — | — | — | |
| Zero-shot CoTModel=GPT-3.5-Turbo, Evaluation Protocol=Zero-shot2024.04 | 93.5 | — | — | — | |
| Least-to-MostModel=GPT-3.5-Turbo, Evaluation Protocol=Zero-shot2024.04 | 93.5 | — | — | — | |
| Zero-shot PS+Model=GPT-3.5-Turbo, Evaluation Protocol=Zero-shot2024.04 | 93 | — | — | — | |
| APEPerturbation=SC2025.06 | 92.7 | — | — | — | |
| PromptAgentPerturbation=EC2025.06 | 92.5 | — | — | — | |
| PromptAgentPerturbation=SC2025.06 | 92.5 | — | — | — | |
| APEPerturbation=WOO2025.06 | 92.3 | — | — | — | |
| SOTA (Fine-tuned)Mode=Fine-tuned2022.06 | 91.2 | — | — | — | |
| CoTPerturbation=EC2025.06 | 90.9 | — | — | — | |
| APEPerturbation=EC2025.06 | 90.9 | — | — | — | |
| RoPPerturbation=HW2025.06 | 90.9 | — | — | — | |
| StandPerturbation=EC2025.06 | 89.2 | — | — | — | |
| RoPPerturbation=WOO2025.06 | 89 | — | — | — | |
| StandPerturbation=WOO2025.06 | 88.8 | — | — | — | |
| CoTPerturbation=SC2025.06 | 88.4 | — | — | — | |
| CoTPerturbation=WOO2025.06 | 88.2 | — | — | — | |
| StandPerturbation=SC2025.06 | 88 | — | — | — | |
| PromptAgentPerturbation=HW2025.06 | 87.8 | — | — | — | |
| Greedy DecodeModel=code-davinci-0022022.06 | 87.5 | — | — | — | |
| CoTPerturbation=HW2025.06 | 87.4 | — | — | — | |
| Self-ConsistencyModel=text-davinci-0022022.06 | 87.2 | — | — | — | |
| Auto-CoTPrompting=Automatic Chain-of-Thought2022.10 | 87 | — | — | — | |
| Manual-CoTPrompting=Manual Chain-of-Thought2022.10 | 86.6 | — | — | — | |
| DIVERSEModel=text-davinci-0022022.06 | 86.5 | — | — | — | |
| PromptAgentPerturbation=WOO2025.06 | 85.6 | — | — | — | |
| StandPerturbation=HW2025.06 | 84.1 | — | — | — | |
| LLaMA-7BDistillation type=undistilled, Zero-shot=true2026.03 | 83.5 | — | — | — | |
| ARMADALoss Function=L_euclid, Teacher Model=Stable Diffusion, Student Model=LLaMA-7B, Zero-shot=true2026.03 | 83.3 | — | — | — | |
| ARMADALoss Function=L_cosine, Teacher Model=Stable Diffusion, Student Model=LLaMA-7B, Zero-shot=true2026.03 | 83.1 | — | — | — | |
| ARMADALoss Function=L_elementwise, Teacher Model=Stable Diffusion, Student Model=LLaMA-7B, Zero-shot=true2026.03 | 83.1 | — | — | — | |
| APEPerturbation=HW2025.06 | 83.1 | — | — | — | |
| Few-ShotPrompting=Few-shot2022.10 | 82.7 | — | — | — | |
| Self-ConsistencyModel=PaLM 540B2022.06 | 80.7 | — | — | — | |
| Greedy DecodeModel=PaLM 540B2022.06 | 79.5 | — | — | — | |
| CoReBackbone=GPT-J 12B2022.10 | 79.5 | — | — | — | |
| RoPPerturbation=UIC2025.06 | 79.1 | — | — | — | |
| zero-shot*Backbone=Instruct GPT-3 175B, Standard Prompt=false2022.10 | 78.7 | — | — | — | |
| zero-shot-CoT*Backbone=Instruct GPT-3 175B, Standard Prompt=false2022.10 | 78.7 | — | — | — | |
| Zero-ShotPrompting=Zero-shot2022.10 | 78.7 | — | — | — | |
| Zero-Shot-CoTPrompting=Zero-shot Chain-of-Thought2022.10 | 78.7 | — | — | — | |
| APEPerturbation=UIC2025.06 | 78.7 | — | — | — | |
| zero-shot-CoTBackbone=Instruct GPT-3 175B2022.10 | 78 | — | — | — | |
| Greedy DecodeModel=text-davinci-0022022.06 | 76.9 | — | — | — | |
| zero-shotBackbone=Instruct GPT-3 175B2022.10 | 74.6 | — | — | — | |
| DIVERSEModel=GPT-3 davinci (175B)2022.06 | 65.1 | — | — | — | |
| StandPerturbation=UIC2025.06 | 63.6 | — | — | — | |
| Self-ConsistencyModel=LaMDA 137B2022.06 | 63.1 | — | — | — | |
| CoTPerturbation=UIC2025.06 | 62.2 | — | — | — | |
| Self-ConsistencyModel=GPT-3 davinci (175B)2022.06 | 59.6 | — | — | — | |
| PromptAgentPerturbation=UIC2025.06 | 57.3 | — | — | — | |
| Greedy DecodeModel=LaMDA 137B2022.06 | 56.6 | — | — | — | |
| Greedy DecodeModel=GPT-3 davinci (175B)2022.06 | 38.2 | — | — | — | |
| 0-shot CoTModel=GPT-3 (code-davinci-001), Setting=0-shot, Paths=14, Self-consistency=true2023.05 | — | — | — | 48.8 | |
| 0-shot CoTModel=PaLM-62B, Setting=0-shot, # Paths=142023.05 | — | — | — | 74.4 | |
| 0-shot CoTModel=PaLM-540B, Setting=0-shot, # Paths=142023.05 | — | — | — | 88.6 | |
| 5-shot CoTModel=GPT-3 (code-davinci-001), Setting=5-shot, Paths=14, Self-consistency=true2023.05 | — | — | — | 65.9 | |
| 5-shot CoTModel=PaLM-62B, Setting=5-shot, # Paths=142023.05 | — | — | — | 79.8 | |
| 5-shot CoTModel=PaLM-540B, Setting=5-shot, # Paths=142023.05 | — | — | — | 89.2 | |
| Auto-CoTModel=GPT-3 (code-davinci-001), Setting=0-shot, Paths=14, Self-consistency=true2023.05 | — | — | — | 58.1 | |
| Auto-CoTModel=PaLM-62B, Setting=0-shot, # Paths=142023.05 | — | — | — | 77.8 | |
| Auto-CoTModel=PaLM-540B, Setting=0-shot, # Paths=142023.05 | — | — | — | 85.6 | |
| COSPModel=GPT-3 (code-davinci-001), Setting=0-shot, Paths=7+7, Self-consistency=true2023.05 | — | — | — | 64.8 | |
| COSPModel=PaLM-62B, Setting=0-shot, # Paths=7+72023.05 | — | — | — | 78.7 | |
| COSPModel=PaLM-540B, Setting=0-shot, # Paths=7+72023.05 | — | — | — | 90.4 | |
| zero-shotReasoning Method=Zero-shot2022.05 | — | 74.6 | 78.7 | — | |
| zero-shot-cotReasoning Method=Zero-shot-CoT2022.05 | — | 78 | 78.7 | — |