List your company
ResearchFormerly WizWand

Question Answering on StrategyQA

94.4AccuracyOracle
59.35268.45177.5586.649Oct 5, 2022May 8, 2023Dec 10, 2023Jul 13, 2024Feb 14, 2025Sep 18, 2025Apr 22, 2026
Updated 6mo ago

Evaluation results

MethodLinks
OracleBackbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=Oracle2023.0494.4
H2OKV cache budget=5122025.1289
StreamingLLMKV cache budget=3842025.1289
H2OKV cache budget=3842025.1288
KnormKV cache budget=3842025.1288
SABABackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=9.82026.0487.4
H2OKV cache budget=2562025.1287
KnormKV cache budget=5122025.1287
KnormKV cache budget=2562025.1285
StreamingLLMKV cache budget=5122025.1285
SELF-DISC.Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=4.82026.0485
GoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=35.52026.0484.4
S^2RBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=18.52026.0483.1
Full2025.1283
SC(k=5)Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=12.12026.0482.4
LLaMA-3.1-70BApproach=CoT, Learning Paradigm=Vanilla2024.1182.2
H2OKV cache budget=1282025.1281
SnapKVKV cache budget=3842025.1281
AdaptiveLM=Qwen2.5-32B, tau=1, Auxiliary Hyperparameter=0.012025.1180.99
GPT-4oApproach=CoT, Learning Paradigm=Vanilla2024.1180.8
AdaptiveLM=Qwen2.5-32B, tau=Avg, Auxiliary Hyperparameter=varies2025.1180.7
Entropy Equilibrium Sampling (EES)LM=Qwen2.5-32B, tau=12025.1180.23
LLaMA-3.1-70BApproach=Zero-shot, Learning Paradigm=Vanilla2024.1179.6
Self-RefineBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=6.22026.0479.1
RKVKV cache budget=2562025.1279
RKVKV cache budget=5122025.1279
Qwen 2.5-7BApproach=Ours, Learning Paradigm=Fine-tuned2024.1178.6
CoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=2.42026.0478.3
LLaMA-3.1-8BApproach=Ours, Learning Paradigm=Fine-tuned2024.1178
SnapKVKV cache budget=1282025.1278
SnapKVKV cache budget=2562025.1278
Qwen 2.5-7BApproach=Planning-token, Learning Paradigm=Fine-tuned2024.1177.4
RKVKV cache budget=3842025.1277
LUMOS-IQAAgent Model=LLAMA-2-13B, QA Tool=GPT-4, fine-tuning=true2023.1176.7
LLaMA-3.1-8BApproach=Planning-token, Learning Paradigm=Fine-tuned2024.1176.7
Entropy Equilibrium Sampling (EES)LM=Qwen2.5-7B, tau=0.52025.1176.33
SnapKVKV cache budget=5122025.1276
StreamingLLMKV cache budget=2562025.1276
Temperature SamplingLM=Qwen2.5-7B, tau=0.52025.1175.75
MirostatLM=Llama3.1-8B, tau=0.8, Auxiliary Hyperparameter=32025.1175.46
Top-pLM=Qwen2.5-7B, tau=0.5, Auxiliary Hyperparameter=0.92025.1174.79
Entropy Equilibrium Sampling (EES)LM=Llama3.1-8B, tau=0.82025.1174.5
PROMPTEDPrompting Technique=PROMPTED2023.1074
LLaMA-2-13BApproach=Ours, Learning Paradigm=Fine-tuned2024.1173.9
MCRBackbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=MCR (5-chains)2023.0473.6
LUMOS-IQAAgent Model=LLAMA-2-7B, QA Tool=GPT-4, fine-tuning=true2023.1172.4
SC@5Backbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=SC@52023.0472.2
DirectBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=1.02026.0472.1
Zero-ShotPrompting Technique=Zero-Shot2023.1072
Zero-Shot CoTPrompting Technique=Zero-Shot Chain-of-Thought2023.1071.6
SC@3Backbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=SC@32023.0471.5
LLaMA-2-13BApproach=Planning-token, Learning Paradigm=Fine-tuned2024.1171.5
LLaMA-2-7BApproach=Ours, Learning Paradigm=Fine-tuned2024.1170.6
LLaMA-3.1-8BApproach=LoRA, Learning Paradigm=Fine-tuned2024.1170.1
SCRBackbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=SCR (1-chain)2023.0470
GPT-4oApproach=Zero-shot, Learning Paradigm=Vanilla2024.1169.9
Qwen 2.5-7BApproach=CoT, Learning Paradigm=Vanilla2024.1169.6
LLaMA-2-13BApproach=LoRA, Learning Paradigm=Fine-tuned2024.1169.6
LLaMA-3.1-8BApproach=CoT, Learning Paradigm=Vanilla2024.1169.4
SABackbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=SA2023.0469.3
Qwen 2.5-7BApproach=LoRA, Learning Paradigm=Fine-tuned2024.1169.1
CRITICBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=29.22026.0468.1
Output RefinementPrompting Technique=Output Refinement2023.1068
ShadowKV2025.1268
LatentMemMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0267.89
LatentMemMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0266.83
ReWOOAgent Model=GPT-3.5-turbo, QA Tool=GPT-3.5-turbo, fine-tuning=false2023.1166.6
LatentMemMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0266.42
OAgentMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0265.94
LLaMA-3.1-8BApproach=Zero-shot, Learning Paradigm=Vanilla2024.1165.9
LUMOS-IQAAgent Model=LLAMA-2-7B, QA Tool=GPT-3.5-turbo, fine-tuning=true2023.1165.7
LUMOS-IQAAgent Model=LLAMA-2-13B, QA Tool=GPT-3.5-turbo, fine-tuning=true2023.1165.3
LatentMemMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0265.28
GenerativeMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0264.85
ReAcTAgent Model=GPT-3.5-turbo, QA Tool=GPT-3.5-turbo, fine-tuning=false2023.1164.6
G-MemoryMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0264.2
PaLMParameters=540B2022.1064
Qwen 2.5-7BApproach=Zero-shot, Learning Paradigm=Vanilla2024.1164
GenerativeMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0263.97
OAgentMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0263.76
LLaMA-2-7BApproach=Planning-token, Learning Paradigm=Fine-tuned2024.1163.5
OAgentMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0263.25
No-memoryMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0262.94
MetaGPTMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0262.88
MetaGPTMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0262.66
MetaGPTMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0262.66
G-MemoryMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0262.23
Prev 8-shot CoTModel=GPT-3 (code-davinci-001), Setting=Prev, Paths=40, Source=Wang et al. (2022a)2023.0561.7
VoyagerMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0261.57
No-memoryMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0261.5
OAgentMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0261.48
VoyagerMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0261.35
VoyagerMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0261.35
G-MemoryMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0261.35
LLaMA-2-13BApproach=Zero-shot, Learning Paradigm=Vanilla2024.1161.3
VoyagerMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0261.28
LLaMA-2-7BApproach=LoRA, Learning Paradigm=Fine-tuned2024.1161.2
G-MemoryMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.0261.14
GenerativeMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.0260.8
LLaMA-2-7BApproach=Zero-shot, Learning Paradigm=Vanilla2024.1160.7