Arithmetic Reasoning on AQuA (test)
74.63AccuracySGE
Evaluation Results
| Method | Links | |
|---|---|---|
| SGEModel=GPT-4, Tool=Code Interpreter2024.05 | 74.63 | |
| Ours (Natural Program (NP), No Verification)LLM=GPT-3.5-turbo, Prompting Format=Natural Program (NP), Verification Method=None, Voting Strategy=Majority Voting, k=102023.06 | 70.34 | |
| CoT + VotingLLM=GPT-3.5-turbo, Prompting Format=Chain-of-Thought (CoT), Voting Strategy=Majority Voting, k=102023.06 | 70.18 | |
| Decomp PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 69.99 | |
| Refine PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 69.68 | |
| CoT PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 69.57 | |
| Ours (NP + Deductive Verification + UPV)LLM=GPT-3.5-turbo, Prompting Format=Natural Program (NP), Verification Method=Deductive Verification, Voting Strategy=UPV, k=102023.06 | 69.49 | |
| IO PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 67.3 | |
| Faithful CoT + VotingLLM=GPT-3.5-turbo, Prompting Format=Faithful CoT, Voting Strategy=Majority Voting, k=102023.06 | 61.8 | |
| LLMBOOSTBackbone=Qwen-2.5-7B, Params=3 × 7B2025.12 | 61.8 | |
| LLMBOOSTBackbone=Qwen-2.5-3B, Params=3 × 3B2025.12 | 61.2 | |
| voteBackbone=Qwen-2.5-7B, Params=3 × 7B2025.12 | 60.9 | |
| UNITEBackbone=Qwen-2.5-7B, Params=3 × 7B2025.12 | 60.9 | |
| LLMBOOSTBackbone=Qwen-2.5-3B, Params=2 × 3B2025.12 | 60.2 | |
| LLMBOOSTBackbone=Qwen-2.5-7B, Params=2 × 7B2025.12 | 59.8 | |
| voteBackbone=Qwen-2.5-3B, Params=3 × 3B2025.12 | 59.7 | |
| UNITEBackbone=Qwen-2.5-7B, Params=2 × 7B2025.12 | 59.5 | |
| UNITEBackbone=Qwen-2.5-3B, Params=3 × 3B2025.12 | 59.3 | |
| UNITEBackbone=Qwen-2.5-3B, Params=2 × 3B2025.12 | 58.5 | |
| voteBackbone=Qwen-2.5-7B, Params=2 × 7B2025.12 | 57.8 | |
| T-copilotBackbone=Qwen-2.5-7B, Params=2 × 7B2025.12 | 56.9 | |
| Qwen-2.5-7BBackbone=Qwen-2.5-7B, Params=1 × 7B2025.12 | 56.8 | |
| voteBackbone=Qwen-2.5-3B, Params=2 × 3B2025.12 | 54.2 | |
| Qwen-2.5-3BBackbone=Qwen-2.5-3B, Params=1 × 3B2025.12 | 54.1 | |
| T-copilotBackbone=Qwen-2.5-3B, Params=2 × 3B2025.12 | 53.9 | |
| LLMBOOSTBackbone=Llama-3.1-8B, Params=3 × 8B2025.12 | 47.6 | |
| LLMBOOSTBackbone=Llama-3.1-8B, Params=2 × 8B2025.12 | 46.8 | |
| voteBackbone=Llama-3.1-8B, Params=2 × 8B2025.12 | 44.4 | |
| voteBackbone=Llama-3.1-8B, Params=3 × 8B2025.12 | 43.9 | |
| UNITEBackbone=Llama-3.1-8B, Params=3 × 8B2025.12 | 43.3 | |
| T-copilotBackbone=Llama-3.1-8B, Params=2 × 8B2025.12 | 42.8 | |
| UNITEBackbone=Llama-3.1-8B, Params=2 × 8B2025.12 | 42.8 | |
| Llama-3.1-8BBackbone=Llama-3.1-8B, Params=1 × 8B2025.12 | 42.3 | |
| LLMBOOSTBackbone=LLama-3.2-3B, Params=2 × 3B2025.12 | 35.8 | |
| LLMBOOSTBackbone=LLama-3.2-3B, Params=3 × 3B2025.12 | 35.1 | |
| voteBackbone=LLama-3.2-3B, Params=2 × 3B2025.12 | 34.6 | |
| T-copilotBackbone=LLama-3.2-3B, Params=2 × 3B2025.12 | 33.3 | |
| voteBackbone=LLama-3.2-3B, Params=3 × 3B2025.12 | 32.9 | |
| UNITEBackbone=LLama-3.2-3B, Params=3 × 3B2025.12 | 32.5 | |
| LLama-3.2-3BBackbone=LLama-3.2-3B, Params=1 × 3B2025.12 | 31.9 | |
| UNITEBackbone=LLama-3.2-3B, Params=2 × 3B2025.12 | 31.1 | |
| REFTBackbone=LLaMA3-8B2024.10 | 27.5 | |
| DoRA + MoDEKBackbone=LLaMA3-8B2024.10 | 27.4 | |
| RoAd2Model=7B, #Params.=0.04%2024.08 | 26.8 | |
| DoRABackbone=LLaMA3-8B2024.10 | 26.8 | |
| RoAd1Model=7B, #Params.=0.02%2024.08 | 26.4 | |
| LoRAALL + MoDEKBackbone=LLaMA3-8B2024.10 | 26.2 | |
| RoAd2Model=13B, #Params.=0.03%2024.08 | 26 | |
| REFTBackbone=LLaMA2-7B2024.10 | 25.8 | |
| DoRA + MoDEKBackbone=LLaMA2-7B2024.10 | 25.8 | |
| LoRAALLBackbone=LLaMA3-8B2024.10 | 25.6 | |
| LoRAALL + MoDEKBackbone=LLaMA2-7B2024.10 | 25.5 | |
| LoRA¬K + MoDEK (+0.04%)Backbone=LLaMA3-8B2024.10 | 25.5 | |
| RoAd4Model=13B, #Params.=0.07%2024.08 | 25.2 | |
| LoRA¬KBackbone=LLaMA3-8B2024.10 | 25 | |
| RoAd4Model=7B, #Params.=0.08%2024.08 | 24.8 | |
| RoAd1Model=13B, #Params.=0.02%2024.08 | 24.8 | |
| DoRABackbone=LLaMA2-7B2024.10 | 24.8 | |
| LoRAALLBackbone=LLaMA2-7B2024.10 | 24.4 | |
| LoRA¬K + MoDEK (+0.04%)Backbone=LLaMA2-7B2024.10 | 24.3 | |
| LoRA¬KBackbone=LLaMA2-7B2024.10 | 24 | |
| LoReFTModel=13B, #Params.=0.03%2024.08 | 23.6 | |
| AdapterSModel=13B, #Params.=0.80%2024.08 | 22 | |
| LoReFTModel=7B, #Params.=0.03%2024.08 | 21.4 | |
| AdapterPModel=13B, #Params.=2.89%2024.08 | 20.5 | |
| (IA)3Model=7B, #Params.=0.02%2024.08 | 19.7 | |
| LORAModel=7B, #Params.=0.83%2024.08 | 18.9 | |
| LORAModel=13B, #Params.=0.67%2024.08 | 18.5 | |
| AdapterPModel=7B, #Params.=3.54%2024.08 | 18.1 | |
| PrefixModel=13B, #Params.=0.03%2024.08 | 15.7 | |
| AdapterSModel=7B, #Params.=0.99%2024.08 | 15 | |
| PrefixModel=7B, #Params.=0.04%2024.08 | 14.2 |