Arithmetic Reasoning on SVAMP
93.7Accuracy (Overall)Automatic Model Selection with LLMs
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Automatic Model Selection with LLMsBackbone=GPT-4, Decoding strategy=Greedy2023.05 | 93.7 | — | — | |
| PALBackbone=GPT-4, Decoding strategy=Greedy2023.05 | 92.2 | — | — | |
| CoTBackbone=GPT-4, Decoding strategy=Greedy2023.05 | 91.9 | — | — | |
| FLAREBackbone=Llama-3-8B, Protocol=ICL2025.02 | 86 | — | — | |
| Automatic Model Selection with LLMsBackbone=ChatGPT, Decoding strategy=Greedy2023.05 | 84.3 | — | — | |
| CoTBackbone=ChatGPT, Decoding strategy=Greedy2023.05 | 83 | — | — | |
| QuaSARBackbone=Llama-3-8B, Protocol=Annotation Tuning2025.02 | 82.6 | — | — | |
| Automatic Model Selection with LLMsBackbone=Codex, Decoding strategy=Greedy2023.05 | 82.2 | — | — | |
| PALBackbone=ChatGPT, Decoding strategy=Greedy2023.05 | 80.3 | — | — | |
| PaLM 540BPrompting=Chain of thought, External Calculator=true2022.01 | 79.8 | — | — | |
| PALBackbone=Codex, Decoding strategy=Greedy2023.05 | 79.6 | — | — | |
| PaLM 540BPrompting=Chain of thought, External Calculator=false2022.01 | 79 | — | — | |
| COTBackbone=Llama-3-8B, Protocol=Annotation Tuning2025.02 | 78.8 | — | — | |
| CoTBackbone=Codex, Decoding strategy=Greedy2023.05 | 77.6 | — | — | |
| QuaSARBackbone=Llama-3-8B, Protocol=ICL2025.02 | 77.3 | — | — | |
| CodexEngine=code-davinci-002, Prompting=Chain of thought, External Calculator=true2022.01 | 77 | — | — | |
| CodexEngine=code-davinci-002, Prompting=Chain of thought, External Calculator=false2022.01 | 76.4 | — | — | |
| COTBackbone=Llama-3-8B, Protocol=ICL2025.02 | 76.3 | — | — | |
| QuaSARBackbone=Qwen2-7B, Protocol=Annotation Tuning2025.02 | 75 | — | — | |
| COTBackbone=Qwen2-7B, Protocol=Annotation Tuning2025.02 | 74.9 | — | — | |
| Llama-3-8BStrategy=Base, Protocol=Annotation Tuning2025.02 | 73.8 | — | — | |
| COTBackbone=Qwen2-7B, Protocol=ICL2025.02 | 73.2 | — | — | |
| GPT-3 175BEngine=text-davinci-002, Prompting=Chain of thought, External Calculator=true2022.01 | 70.3 | — | — | |
| QuaSARBackbone=Qwen2-1.5B, Protocol=Annotation Tuning2025.02 | 70.2 | — | — | |
| Llama-3-8BStrategy=Base, Protocol=ICL2025.02 | 70 | — | — | |
| CodexEngine=code-davinci-002, Prompting=Standard, External Calculator=false2022.01 | 69.9 | — | — | |
| PaLM 540BPrompting=Standard, External Calculator=false2022.01 | 69.4 | — | — | |
| QuaSARBackbone=Qwen2-7B, Protocol=ICL2025.02 | 69.2 | — | — | |
| GPT-3 175BEngine=text-davinci-002, Prompting=Chain of thought, External Calculator=false2022.01 | 68.9 | — | — | |
| Qwen2-7BStrategy=Base, Protocol=Annotation Tuning2025.02 | 67.2 | — | — | |
| Qwen2-7BStrategy=Base, Protocol=ICL2025.02 | 66.9 | — | — | |
| GPT-3 175BEngine=text-davinci-002, Prompting=Standard, External Calculator=false2022.01 | 65.7 | — | — | |
| QuaSARBackbone=Qwen2-1.5B, Protocol=ICL2025.02 | 65.4 | — | — | |
| COTBackbone=Qwen2-1.5B, Protocol=Annotation Tuning2025.02 | 65 | — | — | |
| QuaSARBackbone=Llama-3-1B, Protocol=Annotation Tuning2025.02 | 64.5 | — | — | |
| COTBackbone=Qwen2-1.5B, Protocol=ICL2025.02 | 63.6 | — | — | |
| QuaSARBackbone=Llama-3-1B, Protocol=ICL2025.02 | 60.4 | — | — | |
| Qwen2-1.5BStrategy=Base, Protocol=Annotation Tuning2025.02 | 60 | — | — | |
| COTBackbone=Llama-3-1B, Protocol=Annotation Tuning2025.02 | 59.9 | — | — | |
| Qwen2-1.5BStrategy=Base, Protocol=ICL2025.02 | 59.2 | — | — | |
| COTBackbone=Llama-3-1B, Protocol=ICL2025.02 | 58.2 | — | — | |
| Prior bestPrompting=N/A2022.01 | 57.4 | — | — | |
| Llama-3-1BStrategy=Base, Protocol=Annotation Tuning2025.02 | 50.8 | — | — | |
| Llama-3-1BStrategy=Base, Protocol=ICL2025.02 | 49.5 | — | — | |
| LaMDA 137BPrompting=Chain of thought, External Calculator=true2022.01 | 42.1 | — | — | |
| UL2 20BPrompting=CoT prompting, Calculator=True, Self-consistency=True2022.05 | 41.4 | — | — | |
| LaMDA 137BPrompting=Chain of thought, External Calculator=false2022.01 | 37.5 | — | — | |
| LaMDA 137BPrompting=Standard, External Calculator=false2022.01 | 29.5 | — | — | |
| UL2 20BPrompting=Chain of thought, External Calculator=true2022.01 | 28.3 | — | — | |
| UL2 20BPrompting=CoT prompting, Calculator=True2022.05 | 28.3 | — | — | |
| UL2 20BPrompting=Chain of thought, External Calculator=false2022.01 | 12.5 | — | — | |
| UL2 20BPrompting=CoT prompting2022.05 | 12.5 | — | — | |
| UL2 20BPrompting=Standard, External Calculator=false2022.01 | 10.1 | — | — | |
| UL2 20BPrompting=standard prompting2022.05 | 10.1 | — | — | |
| CRFTBackbone=LLaMA-2-7B, Identify=SAF2025.07 | — | 53.4 | 52.6 | |
| CRFTBackbone=LLaMA-2-7B, Identify=MAF2025.07 | — | 52.4 | 53.4 | |
| CRFTBackbone=LLaMA-2-7B, Identify=SSF2025.07 | — | 52.5 | 53.3 | |
| CRFTBackbone=LLaMA-2-7B, Identify=MSF2025.07 | — | 52.3 | 52.5 | |
| CRFTBackbone=LLaMA-3-8B, Identify=SAF2025.07 | — | 75.5 | 76.1 | |
| CRFTBackbone=LLaMA-3-8B, Identify=MAF2025.07 | — | 77.1 | 77.9 | |
| CRFTBackbone=LLaMA-3-8B, Identify=SSF2025.07 | — | 78 | 78.1 | |
| CRFTBackbone=LLaMA-3-8B, Identify=MSF2025.07 | — | 75.2 | 74.8 | |
| CRFTBackbone=Mistral-7B, Identify=SAF2025.07 | — | 65.9 | 66.2 | |
| CRFTBackbone=Mistral-7B, Identify=MAF2025.07 | — | 66.3 | 65.3 | |
| CRFTBackbone=Mistral-7B, Identify=SSF2025.07 | — | 70.3 | 70.6 | |
| CRFTBackbone=Mistral-7B, Identify=MSF2025.07 | — | 66 | 66.9 | |
| ReFTBackbone=LLaMA-2-7B, Identify=p7+s72025.07 | — | 52.2 | 52.2 | |
| ReFTBackbone=LLaMA-3-8B, Identify=p7+s72025.07 | — | 74.2 | 74.2 | |
| ReFTBackbone=Mistral-7B, Identify=p7+s72025.07 | — | 67.4 | 67.4 |