List your company
ResearchFormerly WizWand

Arithmetic Reasoning on AQuA (test)

74.63AccuracySGE
11.782828.098944.41560.7311Jun 6, 2023Nov 8, 2023Apr 12, 2024Sep 15, 2024Feb 17, 2025Jul 23, 2025Dec 26, 2025
Updated 4mo ago

Evaluation results

MethodLinks
SGEModel=GPT-4, Tool=Code Interpreter2024.0574.63
Ours (Natural Program (NP), No Verification)LLM=GPT-3.5-turbo, Prompting Format=Natural Program (NP), Verification Method=None, Voting Strategy=Majority Voting, k=102023.0670.34
CoT + VotingLLM=GPT-3.5-turbo, Prompting Format=Chain-of-Thought (CoT), Voting Strategy=Majority Voting, k=102023.0670.18
Decomp PromptingModel=GPT-4, Tool=Code Interpreter2024.0569.99
Refine PromptingModel=GPT-4, Tool=Code Interpreter2024.0569.68
CoT PromptingModel=GPT-4, Tool=Code Interpreter2024.0569.57
Ours (NP + Deductive Verification + UPV)LLM=GPT-3.5-turbo, Prompting Format=Natural Program (NP), Verification Method=Deductive Verification, Voting Strategy=UPV, k=102023.0669.49
IO PromptingModel=GPT-4, Tool=Code Interpreter2024.0567.3
Faithful CoT + VotingLLM=GPT-3.5-turbo, Prompting Format=Faithful CoT, Voting Strategy=Majority Voting, k=102023.0661.8
LLMBOOSTBackbone=Qwen-2.5-7B, Params=3 × 7B2025.1261.8
LLMBOOSTBackbone=Qwen-2.5-3B, Params=3 × 3B2025.1261.2
voteBackbone=Qwen-2.5-7B, Params=3 × 7B2025.1260.9
UNITEBackbone=Qwen-2.5-7B, Params=3 × 7B2025.1260.9
LLMBOOSTBackbone=Qwen-2.5-3B, Params=2 × 3B2025.1260.2
LLMBOOSTBackbone=Qwen-2.5-7B, Params=2 × 7B2025.1259.8
voteBackbone=Qwen-2.5-3B, Params=3 × 3B2025.1259.7
UNITEBackbone=Qwen-2.5-7B, Params=2 × 7B2025.1259.5
UNITEBackbone=Qwen-2.5-3B, Params=3 × 3B2025.1259.3
UNITEBackbone=Qwen-2.5-3B, Params=2 × 3B2025.1258.5
voteBackbone=Qwen-2.5-7B, Params=2 × 7B2025.1257.8
T-copilotBackbone=Qwen-2.5-7B, Params=2 × 7B2025.1256.9
Qwen-2.5-7BBackbone=Qwen-2.5-7B, Params=1 × 7B2025.1256.8
voteBackbone=Qwen-2.5-3B, Params=2 × 3B2025.1254.2
Qwen-2.5-3BBackbone=Qwen-2.5-3B, Params=1 × 3B2025.1254.1
T-copilotBackbone=Qwen-2.5-3B, Params=2 × 3B2025.1253.9
LLMBOOSTBackbone=Llama-3.1-8B, Params=3 × 8B2025.1247.6
LLMBOOSTBackbone=Llama-3.1-8B, Params=2 × 8B2025.1246.8
voteBackbone=Llama-3.1-8B, Params=2 × 8B2025.1244.4
voteBackbone=Llama-3.1-8B, Params=3 × 8B2025.1243.9
UNITEBackbone=Llama-3.1-8B, Params=3 × 8B2025.1243.3
T-copilotBackbone=Llama-3.1-8B, Params=2 × 8B2025.1242.8
UNITEBackbone=Llama-3.1-8B, Params=2 × 8B2025.1242.8
Llama-3.1-8BBackbone=Llama-3.1-8B, Params=1 × 8B2025.1242.3
LLMBOOSTBackbone=LLama-3.2-3B, Params=2 × 3B2025.1235.8
LLMBOOSTBackbone=LLama-3.2-3B, Params=3 × 3B2025.1235.1
voteBackbone=LLama-3.2-3B, Params=2 × 3B2025.1234.6
T-copilotBackbone=LLama-3.2-3B, Params=2 × 3B2025.1233.3
voteBackbone=LLama-3.2-3B, Params=3 × 3B2025.1232.9
UNITEBackbone=LLama-3.2-3B, Params=3 × 3B2025.1232.5
LLama-3.2-3BBackbone=LLama-3.2-3B, Params=1 × 3B2025.1231.9
UNITEBackbone=LLama-3.2-3B, Params=2 × 3B2025.1231.1
REFTBackbone=LLaMA3-8B2024.1027.5
DoRA + MoDEKBackbone=LLaMA3-8B2024.1027.4
RoAd2Model=7B, #Params.=0.04%2024.0826.8
DoRABackbone=LLaMA3-8B2024.1026.8
RoAd1Model=7B, #Params.=0.02%2024.0826.4
LoRAALL + MoDEKBackbone=LLaMA3-8B2024.1026.2
RoAd2Model=13B, #Params.=0.03%2024.0826
REFTBackbone=LLaMA2-7B2024.1025.8
DoRA + MoDEKBackbone=LLaMA2-7B2024.1025.8
LoRAALLBackbone=LLaMA3-8B2024.1025.6
LoRAALL + MoDEKBackbone=LLaMA2-7B2024.1025.5
LoRA¬K + MoDEK (+0.04%)Backbone=LLaMA3-8B2024.1025.5
RoAd4Model=13B, #Params.=0.07%2024.0825.2
LoRA¬KBackbone=LLaMA3-8B2024.1025
RoAd4Model=7B, #Params.=0.08%2024.0824.8
RoAd1Model=13B, #Params.=0.02%2024.0824.8
DoRABackbone=LLaMA2-7B2024.1024.8
LoRAALLBackbone=LLaMA2-7B2024.1024.4
LoRA¬K + MoDEK (+0.04%)Backbone=LLaMA2-7B2024.1024.3
LoRA¬KBackbone=LLaMA2-7B2024.1024
LoReFTModel=13B, #Params.=0.03%2024.0823.6
AdapterSModel=13B, #Params.=0.80%2024.0822
LoReFTModel=7B, #Params.=0.03%2024.0821.4
AdapterPModel=13B, #Params.=2.89%2024.0820.5
(IA)3Model=7B, #Params.=0.02%2024.0819.7
LORAModel=7B, #Params.=0.83%2024.0818.9
LORAModel=13B, #Params.=0.67%2024.0818.5
AdapterPModel=7B, #Params.=3.54%2024.0818.1
PrefixModel=13B, #Params.=0.03%2024.0815.7
AdapterSModel=7B, #Params.=0.99%2024.0815
PrefixModel=7B, #Params.=0.04%2024.0814.2