List your company
ResearchFormerly WizWand

Multi-hop Reasoning on StrategyQA

95.6AccuracyOpenMath2-Llama3.1-70B*
26.64844.54962.4580.351Nov 15, 2023Apr 17, 2024Sep 18, 2024Feb 19, 2025Jul 23, 2025Dec 24, 2025May 28, 2026
Updated 5mo ago

Evaluation results

MethodLinks
OpenMath2-Llama3.1-70B*Model Series=OpenMath2, Base=Llama-3.1-70B, Source=Shen et al. (2025)2025.0295.6
Qwen2.5-Math-72B-InstructModel Series=Qwen2.5-Math, Size=72B2025.0294.3
Qwen-2-72B-Instruct#Tok=800, TFLOPs=115.2, Lat. (s)=10.5, GPU DRAM Usage (GB)=> 145.02026.0592.5
Qwen2.5-Math-7B-S2R-ORLModel Series=Qwen2.5-Math, Size=7B, RL Type=Outcome-level RL (ORL)2025.0290.8
Llama-3.1-70B-Instruct*Model Series=Llama-3.1, Size=70B, Source=Shen et al. (2025)2025.0288.8
Qwen2.5-Math-7B-S2R-BIModel Series=Qwen2.5-Math, Size=7B, RL Type=Binary Reward (BI)2025.0288.7
QwQ-32B-Preview*Model Series=QwQ, Size=32B, Source=Shen et al. (2025)2025.0288.2
DynaGraph (Ours, 8B)#Tok=1,220, TFLOPs=19.5, Lat. (s)=15.3, GPU DRAM Usage (GB)=16.6 (O(1))2026.0587.6
Reflexion#Tok=3,890, TFLOPs=62.2, Lat. (s)=44.5, GPU DRAM Usage (GB)=16.52026.0586.2
Standard ToT#Tok=3,400, TFLOPs=54.4, Lat. (s)=39.5, GPU DRAM Usage (GB)=16.52026.0585.8
ReAct#Tok=2,480, TFLOPs=39.7, Lat. (s)=30.5, GPU DRAM Usage (GB)=16.52026.0584.5
Multi-Agent (3 × 8B)#Tok=1,520, TFLOPs=24.3, Lat. (s)=17.5, GPU DRAM Usage (GB)=> 49.52026.0584
StrategyLLM-SCBackbone=Meta-Llama-3-70B-Instruct2023.1183.5
Self-Consistency (k=5)#Tok=1,700, TFLOPs=27.2, Lat. (s)=19.8, GPU DRAM Usage (GB)=16.52026.0583.3
StrategyLLMBackbone=Meta-Llama-3-70B-Instruct2023.1182
CoT-SCBackbone=Meta-Llama-3-70B-Instruct2023.1181.5
Qwen2.5-Math-7B-InstructModel Series=Qwen2.5-Math, Size=7B2025.0281.2
CoTBackbone=Meta-Llama-3-70B-Instruct2023.1180.5
ShadowKVBackbone=DeepSeek-R1-Distill-Llama-8B2025.1280
Eurus-2-7B-PRIMEModel Series=Eurus-2, Size=7B2025.0279
H2OBackbone=DeepSeek-R1-Distill-Llama-8B, Cache Budget=5122025.1279
SolutionLLMBackbone=Meta-Llama-3-70B-Instruct2023.1179
Standard CoT#Tok=650, TFLOPs=10.4, Lat. (s)=8.0, GPU DRAM Usage (GB)=16.52026.0578.4
StrategyLLM-SCBackbone=Mixtral-8x22B-Instruct-v0.12023.1177
StrategyLLMBackbone=Mixtral-8x22B-Instruct-v0.12023.1176.5
StrategyLLM-SCBackbone=Mixtral-8x7B-Instruct-v0.12023.1175
CoT-SCBackbone=Mixtral-8x22B-Instruct-v0.12023.1175
FullBackbone=DeepSeek-R1-Distill-Llama-8B2025.1274
StrategyLLMBackbone=Meta-Llama-3-8B-Instruct2023.1174
StrategyLLM-SCBackbone=Meta-Llama-3-8B-Instruct2023.1174
CoT-SCBackbone=Mixtral-8x7B-Instruct-v0.12023.1173.5
StrategyLLMBackbone=Mixtral-8x7B-Instruct-v0.12023.1173.5
SolutionLLMBackbone=Mixtral-8x22B-Instruct-v0.12023.1172
CoTBackbone=Mixtral-8x22B-Instruct-v0.12023.1172
CoT-SCBackbone=Meta-Llama-3-8B-Instruct2023.1171
Standard Prompting#Tok=210, TFLOPs=3.4, Lat. (s)=2.5, GPU DRAM Usage (GB)=16.52026.0565.2
SolutionLLMBackbone=Meta-Llama-3-8B-Instruct2023.1164
CoTBackbone=Mixtral-8x7B-Instruct-v0.12023.1163
LoTVerifier Training Dataset=AQuA2025.0362.3
Qwen2.5-Math-7BModel Series=Qwen2.5-Math, Size=7B2025.0261.1
CoTBackbone=Meta-Llama-3-8B-Instruct2023.1161
SolutionLLMBackbone=Mixtral-8x7B-Instruct-v0.12023.1161
LoTVerifier Training Dataset=MMLU2025.0353
LoTVerifier Training Dataset=CommonSenseQA2025.0353
LoTVerifier Training Dataset=StrategyQA2025.0343
Thoughts-as-PlanningAvg Edits=472026.0433.9
CoTGenAvg Edits=1802026.0431.5
RLCoTAvg Edits=1502026.0430.9
SoftCoT2026.0430.6
Manual CoT2026.0429.3