Arithmetic Reasoning on MultiArith (test)
99.5AccuracyLoRAALL
Evaluation results
| Method | Links | |
|---|---|---|
| LoRAALLBackbone=LLaMA3-8B2024.10 | 99.5 | |
| LoRAALL + MoDEKBackbone=LLaMA3-8B2024.10 | 99.5 | |
| PaLMPrompting Strategy=Self-Consistency, Scenario=Scenario 32023.11 | 99.3 | |
| GPT-3.5-turboPrompting Strategy=Self-Consistency, Scenario=Scenario 32023.11 | 99.2 | |
| CoT + PALPrompting=Hybrid CoT + PAL, Model=GPT-4-0613, Shots=4-shot, Decoding Strategy=Greedy decoding2024.05 | 99 | |
| Self-AgreementModel=GPT-3.5-turbo, Scenario=Scenario 32023.11 | 99 | |
| DoRA + MoDEKBackbone=LLaMA3-8B2024.10 | 99 | |
| SIGMAMulti-agent collaboration=full support, Inter-agent relations=full support, Handling conflicting signals=full support2026.05 | 98.87 | |
| MaASParadigm=Automated (Probabilistic Supernet), Backbone=GPT-4o-mini2026.04 | 98.8 | |
| LoRA¬KBackbone=LLaMA3-8B2024.10 | 98.8 | |
| DoRABackbone=LLaMA3-8B2024.10 | 98.8 | |
| LoRA¬K + MoDEK (+0.04%)Backbone=LLaMA3-8B2024.10 | 98.8 | |
| G-DesignerMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 98.72 | |
| SWIFTParadigm=Proposed, Backbone=GPT-4o-mini2026.04 | 98.54 | |
| GPTSwarmMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 98.51 | |
| PALPrompting=Program-Aided Language Models, Model=GPT-4-0613, Shots=4-shot, Decoding Strategy=Greedy decoding2024.05 | 98.5 | |
| PHPMulti-agent collaboration=full support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 98.5 | |
| Complete GraphMulti-agent collaboration=full support, Inter-agent relations=full support, Handling conflicting signals=no support2026.05 | 98.5 | |
| SCMulti-agent collaboration=no support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 98.45 | |
| Random GraphMulti-agent collaboration=full support, Inter-agent relations=full support, Handling conflicting signals=no support2026.05 | 98.43 | |
| Self-MoAMulti-agent collaboration=full support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 98.42 | |
| ComplexCoTMulti-agent collaboration=no support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 98.23 | |
| GoAMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 98.23 | |
| MoAMulti-agent collaboration=full support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 98.22 | |
| GPT-3.5-turboPrompting Strategy=Few-Shot CoT, Scenario=Scenario 32023.11 | 98.2 | |
| GPT-3.5-turboPrompting Strategy=USC, Scenario=Scenario 32023.11 | 98.2 | |
| CoT + Skill-BasedPrompting=Chain-of-Thought with Skill-Based exemplar selection, Model=GPT-4-0613, Shots=4-shot, Decoding Strategy=Greedy decoding2024.05 | 98.17 | |
| AutoGenMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 98.12 | |
| CoTPrompting=Chain-of-Thought, Model=GPT-4-0613, Shots=4-shot, Decoding Strategy=Greedy decoding2024.05 | 98 | |
| CoTMulti-agent collaboration=no support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 97.81 | |
| AgentSquareParadigm=Automated, Backbone=GPT-4o-mini2026.04 | 97.77 | |
| DyLANMulti-agent collaboration=full support, Inter-agent relations=full support2026.05 | 97.54 | |
| CoReBackbone=GPT-J 12B, Evaluation Protocol=Zero-shot2022.10 | 97.5 | |
| AgentVerseParadigm=Hand-craft, Backbone=GPT-4o-mini2026.04 | 97.5 | |
| MultiPersonaParadigm=Hand-craft, Backbone=GPT-4o-mini2026.04 | 97.49 | |
| VanillaMulti-agent collaboration=no support, Inter-agent relations=no support, Handling conflicting signals=no support2026.05 | 97.42 | |
| LLM-DebateParadigm=Hand-craft, Backbone=GPT-4o-mini2026.04 | 97.33 | |
| LLM-BlenderParadigm=Hand-craft, Backbone=GPT-4o-mini2026.04 | 97.29 | |
| DyLANParadigm=Hand-craft, Backbone=GPT-4o-mini2026.04 | 97.12 | |
| CoTTest Engine=Phi-3.5-Mini-Instruct2026.05 | 97 | |
| REFTBackbone=LLaMA3-8B2024.10 | 97 | |
| VanillaParadigm=Single, Backbone=GPT-4o-mini2026.04 | 96.85 | |
| GPTSwarmParadigm=Automated, Backbone=GPT-4o-mini2026.04 | 96.79 | |
| ComplexCoTParadigm=Single, Backbone=GPT-4o-mini2026.04 | 96.7 | |
| TextRegTest Engine=Llama-3.1-8B-Instruct2026.05 | 96.7 | |
| SC (CoT×5)Paradigm=Single, Backbone=GPT-4o-mini2026.04 | 96.58 | |
| TextRegTest Engine=Qwen2-7B-Instruct2026.05 | 96.5 | |
| AutoAgentsParadigm=Automated, Backbone=GPT-4o-mini2026.04 | 96.42 | |
| CoTParadigm=Single, Backbone=GPT-4o-mini2026.04 | 96.31 | |
| AFlowParadigm=Automated, Backbone=GPT-4o-mini2026.04 | 96.22 | |
| REVOLVETest Engine=Qwen2-7B-Instruct2026.05 | 96.2 | |
| LT-Tuning + AdapterBackbone=Llama-3.1-8B, Fine-tuning set=GSM8K training set, Adapter usage=true2026.02 | 96.1 | |
| MacNetParadigm=Hand-craft, Backbone=GPT-4o-mini2026.04 | 96.03 | |
| ADASParadigm=Automated, Backbone=GPT-4o-mini2026.04 | 96.02 | |
| CoTTest Engine=Llama-3.1-8B-Instruct2026.05 | 96 | |
| TextGradTest Engine=Llama-3.1-8B-Instruct2026.05 | 96 | |
| TextGradTest Engine=Qwen2-7B-Instruct2026.05 | 95.8 | |
| REVOLVETest Engine=Llama-3.1-8B-Instruct2026.05 | 95.3 | |
| REVOLVETest Engine=Phi-3.5-Mini-Instruct2026.05 | 95.1 | |
| TextRegTest Engine=Llama-3-8B-Instruct2026.05 | 94.9 | |
| PaLMPrompting Strategy=Few-Shot CoT, Scenario=Scenario 32023.11 | 94.7 | |
| CoTTest Engine=Qwen2-7B-Instruct2026.05 | 94.7 | |
| CoTTest Engine=Llama-3-8B-Instruct2026.05 | 94.7 | |
| TextGradTest Engine=Llama-3-8B-Instruct2026.05 | 94.7 | |
| REVOLVETest Engine=Llama-3-8B-Instruct2026.05 | 94.7 | |
| TextRegTest Engine=Phi-3.5-Mini-Instruct2026.05 | 94.7 | |
| LT-TuningBackbone=Llama-3.1-8B, Fine-tuning set=GSM8K training set, Adapter usage=false2026.02 | 92.8 | |
| Self-ConsistencyBackbone=GPT-J 6B, Evaluation Protocol=Zero-shot2022.10 | 92.3 | |
| PE2Task Model=text-davinci-003, Proposal Model=GPT-42023.11 | 92.3 | |
| Iterative APETask Model=text-davinci-003, Proposal Model=GPT-42023.11 | 88.5 | |
| APOTask Model=text-davinci-003, Proposal Model=GPT-42023.11 | 88.5 | |
| Fisher-IntDim-G + SFTBackbone=Qwen1.5-MoE-A2.7B, MoE compression ratio (p)=50%, Supervised Fine-Tuning (SFT)=true2026.06 | 88 | |
| Base + SFTBackbone=Qwen1.5-MoE-A2.7B, MoE compression ratio (p)=0%, Supervised Fine-Tuning (SFT)=true2026.06 | 87.8 | |
| APETask Model=text-davinci-003, Context=Reproduced2023.11 | 87 | |
| TextGradTest Engine=Phi-3.5-Mini-Instruct2026.05 | 86.8 | |
| Zero-shot CoTTask Model=text-davinci-003, Context=Reproduced2023.11 | 86 | |
| Soft-ThinkingBackbone=Llama-3.1-8B, Fine-tuning set=GSM8K training set, Adapter usage=false2026.02 | 85 | |
| Expert-level Fisher + SFTBackbone=Qwen1.5-MoE-A2.7B, MoE compression ratio (p)=50%, Supervised Fine-Tuning (SFT)=true2026.06 | 84.8 | |
| DoRA + MoDEKBackbone=LLaMA2-7B2024.10 | 84.6 | |
| DoRABackbone=LLaMA2-7B2024.10 | 84.5 | |
| LoRAALL + MoDEKBackbone=LLaMA2-7B2024.10 | 84.3 | |
| LoRAALLBackbone=LLaMA2-7B2024.10 | 84.2 | |
| LoRA¬K + MoDEK (+0.04%)Backbone=LLaMA2-7B2024.10 | 82.2 | |
| APETask Model=text-davinci-002, Proposal Model=text-davinci-002, Context=Reported by Zhou et al. (2023b)2023.11 | 82 | |
| MoE comp. (Fisher) + SFTBackbone=Qwen1.5-MoE-A2.7B, MoE compression ratio (p)=50%, Supervised Fine-Tuning (SFT)=true2026.06 | 81.8 | |
| LoRA¬KBackbone=LLaMA2-7B2024.10 | 81.4 | |
| REFTBackbone=LLaMA2-7B2024.10 | 80.5 | |
| Zero-shot CoTTask Model=text-davinci-002, Context=Reported by Zhou et al. (2023b)2023.11 | 78.7 | |
| Explicit CoTBackbone=Llama-3.1-8B, Fine-tuning set=GSM8K training set, Adapter usage=false2026.02 | 78.3 | |
| Generator onlyBackbone=GPT-J 6B, Evaluation Protocol=Zero-shot2022.10 | 77.3 | |
| LaMDAPrompting Strategy=Self-Consistency, Scenario=Scenario 32023.11 | 75.7 | |
| SoftCoTBackbone=Llama-3.1-8B, Fine-tuning set=GSM8K training set, Adapter usage=false2026.02 | 74.4 | |
| PE2Final Prompt=Focus on accurately calculating totals and differences, considering factors like item conditions or groupings for precision. Round only if necessary, when dealing with practical fractions., Task Model=Mistral-7B-Instruct-v0.2, Prompt Proposal Model=gpt-4-turbo2023.11 | 74.3 | |
| Iterative APEFinal Prompt=Proceed gradually, one step at a time., Task Model=Mistral-7B-Instruct-v0.2, Prompt Proposal Model=gpt-4-turbo2023.11 | 73.5 | |
| APOFinal Prompt=Calculate the answer using arithmetic. Round down where necessary. Correct any logical errors in reasoning. Provide the exact number., Task Model=Mistral-7B-Instruct-v0.2, Prompt Proposal Model=gpt-4-turbo2023.11 | 73.5 | |
| Zero-shot CoTFinal Prompt=Let's think step by step., Task Model=Mistral-7B-Instruct-v0.2, Prompt Proposal Model=gpt-4-turbo2023.11 | 71.5 | |
| SemCoTBackbone=Llama-3.1-8B, Fine-tuning set=GSM8K training set, Adapter usage=false2026.02 | 67.8 | |
| LT-TuningBackbone=Llama-3.2-3B, Fine-tuning set=GSM8K training set, Adapter usage=false2026.02 | 64.4 | |
| CoconutBackbone=Llama-3.2-3B, Fine-tuning set=GSM8K training set, Adapter usage=false2026.02 | 63.3 | |
| Relevance and LCA operation classifierEvaluation Protocol=Fine-tune2022.10 | 60.5 |