List your company
ResearchFormerly WizWand

Question Answering on OpenBookQA

94.4AccuracyLMSI
71.31277.30683.389.294May 2, 2020Apr 20, 2021Apr 9, 2022Mar 29, 2023Mar 17, 2024Mar 6, 2025Feb 23, 2026
Updated 8mo ago

Evaluation results

MethodLinks
LMSIPrompting Method=Self-Consistency2022.1094.4——
LMSIPrompting Method=CoT-Prompting2022.1093——
LMSIPrompting Method=Standard-Prompting2022.1092——
Wang et al. (2022a)description=Previous SOTA2022.1091——
Qwen3 14B BaseClassifier=Self-labeled2026.0190.8——
Qwen3 14B BaseClassifier=Majority-labeled2026.0190.8——
Qwen3 14BClassifier=Self-labeled2026.0190.4——
Self-ConsistencyLMSI=false2022.1090——
Qwen3 14BClassifier=Majority-labeled2026.0190——
Yi-34Bevaluation=5-shot ICL2024.0989.8——
Yi-34B + RTDevaluation=5-shot2024.0988.8——
Yi-34B + RTDevaluation=zero-shot2024.0988.4——
Self-rethinkingModel=PaLM2, Prompting Strategy=Self-rethinking, k=12024.0387.71——
Self-consistencyModel=PaLM2, Prompting Strategy=Self-consistency, inference times=32024.0387.61——
UNIFIEDQAModel Architecture=T5, Retrieval Context (IR)=Yes, Fine-tuning Protocol=Fine-tuned2020.0587.2——
MixDoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.0486.9——
CoT-PromptingLMSI=false2022.1086.4——
UNIFIEDQAModel Architecture=T5, Retrieval Context (IR)=No, Fine-tuning Protocol=Fine-tuned2020.0586——
Qwen3 8BClassifier=Self-labeled2026.0185.8——
Qwen3 8BClassifier=Majority-labeled2026.0185.6——
LLaMA2-70B + RTDevaluation=5-shot2024.0985.4——
LoRABase Model=LLaMA-3 8B, Trainable Parameters=2.6%2024.0485——
MixLoRABase Model=LLaMA-3 8B, Trainable Parameters=3.0%2024.0484.8——
DoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.0484.5——
Standard-PromptingLMSI=false2022.1084.4——
LLaMA2-70Bevaluation=5-shot ICL2024.0984.4——
Qwen3 8B BaseClassifier=Majority-labeled2026.0184.4——
T5Model Architecture=T5, Retrieval Context (IR)=No, Fine-tuning Protocol=Fine-tuned2020.0584.2——
T5Model Architecture=T5, Retrieval Context (IR)=Yes, Fine-tuning Protocol=Fine-tuned2020.0584.2——
Qwen3 8B BaseClassifier=Self-labeled2026.0184.2——
Yi-34Bevaluation=zero-shot2024.0983.5——
MixDoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.0483.4——
LoRABase Model=LLaMA-2 13B, Trainable Parameters=2.4%2024.0483.2——
MixLoRABase Model=LLaMA-2 13B, Trainable Parameters=2.5%2024.0483——
CoTModel=PaLM2, Prompting Strategy=Chain-of-Thought2024.0382.66——
MixLoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.0481.6——
Qwen3 4B BaseClassifier=Majority-labeled2026.0181.6——
Qwen3 4B BaseClassifier=Self-labeled2026.0181.4——
Standard PromptingModel=PaLM2, Prompting Strategy=Standard Prompting2024.0380.92——
MixDoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.0480.9——
DoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.0480.6——
DoRABase Model=LLaMA-3 8B, Trainable Parameters=2.6%2024.0480.6——
LoRABase Model=LLaMA-2 7B, Trainable Parameters=2.9%2024.0480.4——
KF+SIRModel Architecture=RoBERTa, Retrieval Context (IR)=Yes, Fine-tuning Protocol=standard2020.0580——
DrICLk=50, Backbone=Llama-2-7b-chat-hf2025.0180——
DrICLk=MAX, Backbone=Llama-2-7b-chat-hf2025.0180——
OPT-IML 175BEvaluation protocol=0-shot2022.1279.9——
TS (C)Algorithm Family=Contextual Linear2026.0279.3——
MetaICLk=60, Backbone=Llama-2-7b-chat-hf2025.0179——
MetaICLk=MAX, Backbone=Llama-2-7b-chat-hf2025.0179——
DrICLNumber of shots (k)=102025.0179——
DrICLNumber of shots (k)=302025.0179——
DrICLNumber of shots (k)=402025.0179——
DrICLNumber of shots (k)=502025.0179——
LLaMA3-8B + RTDevaluation=5-shot2024.0978.6——
Qwen3 4BClassifier=Self-labeled2026.0178.4——
MetaICLk=40, Backbone=Llama-2-7b-chat-hf2025.0178——
MetaICLk=50, Backbone=Llama-2-7b-chat-hf2025.0178——
DrICLk=10, Backbone=Llama-2-7b-chat-hf2025.0178——
DrICLNumber of shots (k)=202025.0178——
Qwen3 4BClassifier=Majority-labeled2026.0178——
LLaMA3-8Bevaluation=5-shot ICL2024.0977.5——
FLAN 137BEvaluation protocol=0-shot, Training strategy=leave-one-category-out2022.1277.4——
FLAN 137BEvaluation protocol=few-shot, k=variable, Training strategy=leave-one-category-out2022.1277.2——
MetaICLk=30, Backbone=Llama-2-7b-chat-hf2025.0177——
DrICLk=3, Backbone=Llama-2-7b-chat-hf2025.0177——
DrICLk=5, Backbone=Llama-2-7b-chat-hf2025.0177——
DrICLNumber of shots (k)=32025.0177——
DrICLNumber of shots (k)=52025.0177——
DrICLNumber of shots (k)=602025.0177——
DrICLNumber of shots (k)=702025.0177——
OPT-IML 30BEvaluation protocol=0-shot2022.1276.7——
OPT-IML 175BEvaluation protocol=few-shot, k=52022.1276.5——
FalconModel Size=180B2023.1176.4——
GFLOWPOPrompt LM=Gemma-7B, Target LM=Gemma-7B2026.0276.2——
DrICLk=20, Backbone=Llama-2-7b-chat-hf2025.0176——
DrICLk=30, Backbone=Llama-2-7b-chat-hf2025.0176——
DrICLk=40, Backbone=Llama-2-7b-chat-hf2025.0176——
DrICLk=60, Backbone=Llama-2-7b-chat-hf2025.0176——
DrICLk=70, Backbone=Llama-2-7b-chat-hf2025.0176——
DrICLk=AVG, Backbone=Llama-2-7b-chat-hf2025.0176——
MetaICLNumber of shots (k)=202025.0176——
MetaICLNumber of shots (k)=402025.0176——
MetaICLNumber of shots (k)=502025.0176——
MetaICLNumber of shots (k)=602025.0176——
BaselineFormat=Baseline, Bit width (b)=16.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.0576——
MixLoRABase Model=Gemma 2B, Trainable Parameters=4.3%2024.0475.8——
FreeLB-RoBERTaModel Architecture=RoBERTa, Retrieval Context (IR)=No, Fine-tuning Protocol=standard2020.0575.7——
MixDoRABase Model=Gemma 2B, Trainable Parameters=4.3%2024.0475.4——
MetaICLk=20, Backbone=Llama-2-7b-chat-hf2025.0175——
MetaICLNumber of shots (k)=102025.0175——
MetaICLNumber of shots (k)=702025.0174——
DrICLNumber of shots (k)=02025.0174——
DrICLNumber of shots (k)=12025.0174——
No-RewriteAlgorithm Family=Base2026.0273.5——
MetaICLNumber of shots (k)=52025.0173——
Tensor RMS + CFormat=Tensor RMS + C, Bit width (b)=3.00, Quantization Aware Training (QAT)=true, Backbone=Llama 3.1 8B2025.0572.8——
FLIPPEDModel Size=11B, Evaluation Protocol=Zero-shot2022.1072.54——
STABLEPROMPTPrompt LM=Gemma-7B, Target LM=Gemma-7B2026.0272.2——
Qwen3 1.7B BaseClassifier=Self-labeled2026.0172.2——