Question Answering on StrategyQA
94.4AccuracyOracle
Evaluation Results
| Method | Links | |
|---|---|---|
| OracleBackbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=Oracle2023.04 | 94.4 | |
| H2OKV cache budget=5122025.12 | 89 | |
| StreamingLLMKV cache budget=3842025.12 | 89 | |
| H2OKV cache budget=3842025.12 | 88 | |
| KnormKV cache budget=3842025.12 | 88 | |
| SABABackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=9.82026.04 | 87.4 | |
| H2OKV cache budget=2562025.12 | 87 | |
| KnormKV cache budget=5122025.12 | 87 | |
| KnormKV cache budget=2562025.12 | 85 | |
| StreamingLLMKV cache budget=5122025.12 | 85 | |
| SELF-DISC.Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=4.82026.04 | 85 | |
| GoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=35.52026.04 | 84.4 | |
| S^2RBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=18.52026.04 | 83.1 | |
| Full2025.12 | 83 | |
| SC(k=5)Backbone model=Gemini-1.5-Flash, Normalized inference cost (T)=12.12026.04 | 82.4 | |
| LLaMA-3.1-70BApproach=CoT, Learning Paradigm=Vanilla2024.11 | 82.2 | |
| H2OKV cache budget=1282025.12 | 81 | |
| SnapKVKV cache budget=3842025.12 | 81 | |
| AdaptiveLM=Qwen2.5-32B, tau=1, Auxiliary Hyperparameter=0.012025.11 | 80.99 | |
| GPT-4oApproach=CoT, Learning Paradigm=Vanilla2024.11 | 80.8 | |
| AdaptiveLM=Qwen2.5-32B, tau=Avg, Auxiliary Hyperparameter=varies2025.11 | 80.7 | |
| Entropy Equilibrium Sampling (EES)LM=Qwen2.5-32B, tau=12025.11 | 80.23 | |
| LLaMA-3.1-70BApproach=Zero-shot, Learning Paradigm=Vanilla2024.11 | 79.6 | |
| Self-RefineBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=6.22026.04 | 79.1 | |
| RKVKV cache budget=2562025.12 | 79 | |
| RKVKV cache budget=5122025.12 | 79 | |
| Qwen 2.5-7BApproach=Ours, Learning Paradigm=Fine-tuned2024.11 | 78.6 | |
| CoTBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=2.42026.04 | 78.3 | |
| LLaMA-3.1-8BApproach=Ours, Learning Paradigm=Fine-tuned2024.11 | 78 | |
| SnapKVKV cache budget=1282025.12 | 78 | |
| SnapKVKV cache budget=2562025.12 | 78 | |
| Qwen 2.5-7BApproach=Planning-token, Learning Paradigm=Fine-tuned2024.11 | 77.4 | |
| RKVKV cache budget=3842025.12 | 77 | |
| LUMOS-IQAAgent Model=LLAMA-2-13B, QA Tool=GPT-4, fine-tuning=true2023.11 | 76.7 | |
| LLaMA-3.1-8BApproach=Planning-token, Learning Paradigm=Fine-tuned2024.11 | 76.7 | |
| Entropy Equilibrium Sampling (EES)LM=Qwen2.5-7B, tau=0.52025.11 | 76.33 | |
| SnapKVKV cache budget=5122025.12 | 76 | |
| StreamingLLMKV cache budget=2562025.12 | 76 | |
| Temperature SamplingLM=Qwen2.5-7B, tau=0.52025.11 | 75.75 | |
| MirostatLM=Llama3.1-8B, tau=0.8, Auxiliary Hyperparameter=32025.11 | 75.46 | |
| Top-pLM=Qwen2.5-7B, tau=0.5, Auxiliary Hyperparameter=0.92025.11 | 74.79 | |
| Entropy Equilibrium Sampling (EES)LM=Llama3.1-8B, tau=0.82025.11 | 74.5 | |
| PROMPTEDPrompting Technique=PROMPTED2023.10 | 74 | |
| LLaMA-2-13BApproach=Ours, Learning Paradigm=Fine-tuned2024.11 | 73.9 | |
| MCRBackbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=MCR (5-chains)2023.04 | 73.6 | |
| LUMOS-IQAAgent Model=LLAMA-2-7B, QA Tool=GPT-4, fine-tuning=true2023.11 | 72.4 | |
| SC@5Backbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=SC@52023.04 | 72.2 | |
| DirectBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=1.02026.04 | 72.1 | |
| Zero-ShotPrompting Technique=Zero-Shot2023.10 | 72 | |
| Zero-Shot CoTPrompting Technique=Zero-Shot Chain-of-Thought2023.10 | 71.6 | |
| SC@3Backbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=SC@32023.04 | 71.5 | |
| LLaMA-2-13BApproach=Planning-token, Learning Paradigm=Fine-tuned2024.11 | 71.5 | |
| LLaMA-2-7BApproach=Ours, Learning Paradigm=Fine-tuned2024.11 | 70.6 | |
| LLaMA-3.1-8BApproach=LoRA, Learning Paradigm=Fine-tuned2024.11 | 70.1 | |
| SCRBackbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=SCR (1-chain)2023.04 | 70 | |
| GPT-4oApproach=Zero-shot, Learning Paradigm=Vanilla2024.11 | 69.9 | |
| Qwen 2.5-7BApproach=CoT, Learning Paradigm=Vanilla2024.11 | 69.6 | |
| LLaMA-2-13BApproach=LoRA, Learning Paradigm=Fine-tuned2024.11 | 69.6 | |
| LLaMA-3.1-8BApproach=CoT, Learning Paradigm=Vanilla2024.11 | 69.4 | |
| SABackbone=code-davinci-002, Reasoning Category=implicit, Number of Examples=1000, Reasoning Strategy=SA2023.04 | 69.3 | |
| Qwen 2.5-7BApproach=LoRA, Learning Paradigm=Fine-tuned2024.11 | 69.1 | |
| CRITICBackbone model=Gemini-1.5-Flash, Normalized inference cost (T)=29.22026.04 | 68.1 | |
| Output RefinementPrompting Technique=Output Refinement2023.10 | 68 | |
| ShadowKV2025.12 | 68 | |
| LatentMemMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 67.89 | |
| LatentMemMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 66.83 | |
| ReWOOAgent Model=GPT-3.5-turbo, QA Tool=GPT-3.5-turbo, fine-tuning=false2023.11 | 66.6 | |
| LatentMemMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 66.42 | |
| OAgentMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 65.94 | |
| LLaMA-3.1-8BApproach=Zero-shot, Learning Paradigm=Vanilla2024.11 | 65.9 | |
| LUMOS-IQAAgent Model=LLAMA-2-7B, QA Tool=GPT-3.5-turbo, fine-tuning=true2023.11 | 65.7 | |
| LUMOS-IQAAgent Model=LLAMA-2-13B, QA Tool=GPT-3.5-turbo, fine-tuning=true2023.11 | 65.3 | |
| LatentMemMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 65.28 | |
| GenerativeMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 64.85 | |
| ReAcTAgent Model=GPT-3.5-turbo, QA Tool=GPT-3.5-turbo, fine-tuning=false2023.11 | 64.6 | |
| G-MemoryMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 64.2 | |
| PaLMParameters=540B2022.10 | 64 | |
| Qwen 2.5-7BApproach=Zero-shot, Learning Paradigm=Vanilla2024.11 | 64 | |
| GenerativeMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 63.97 | |
| OAgentMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 63.76 | |
| LLaMA-2-7BApproach=Planning-token, Learning Paradigm=Fine-tuned2024.11 | 63.5 | |
| OAgentMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 63.25 | |
| No-memoryMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 62.94 | |
| MetaGPTMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 62.88 | |
| MetaGPTMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 62.66 | |
| MetaGPTMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 62.66 | |
| G-MemoryMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 62.23 | |
| Prev 8-shot CoTModel=GPT-3 (code-davinci-001), Setting=Prev, Paths=40, Source=Wang et al. (2022a)2023.05 | 61.7 | |
| VoyagerMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 61.57 | |
| No-memoryMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 61.5 | |
| OAgentMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 61.48 | |
| VoyagerMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 61.35 | |
| VoyagerMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 61.35 | |
| G-MemoryMAS Framework=DyLAN, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 61.35 | |
| LLaMA-2-13BApproach=Zero-shot, Learning Paradigm=Vanilla2024.11 | 61.3 | |
| VoyagerMAS Framework=AutoGen, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 61.28 | |
| LLaMA-2-7BApproach=LoRA, Learning Paradigm=Fine-tuned2024.11 | 61.2 | |
| G-MemoryMAS Framework=CAMEL, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-out2026.02 | 61.14 | |
| GenerativeMAS Framework=MacNet, LLM Backbone=Llama-3.1-8B-Instruct, MAS Framework Evaluation Setting=Held-in2026.02 | 60.8 | |
| LLaMA-2-7BApproach=Zero-shot, Learning Paradigm=Vanilla2024.11 | 60.7 |