Question Answering on OpenBookQA (test)
92.4OBQA AccuracyKnowGPT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| KnowGPTCategory=Ours2023.12 | 92.4 | — | |
| GPT-4Category=LLM + Zero-shot2023.12 | 91 | — | |
| MindmapCategory=LLM + KG Prompting2023.12 | 88.2 | — | |
| AristoRoBERTa + GSCadditional_input=science facts2021.10 | 87.4 | — | |
| UnifiedQA(11B)*additional_input=science facts, number of parameters=11B2021.10 | 87.2 | — | |
| UnifiedQA*Additional science facts input=true, Model size (parameters)=11B2021.04 | 87.2 | — | |
| CoKCategory=LLM + KG Prompting2023.12 | 86.9 | — | |
| RoGCategory=LLM + KG Prompting2023.12 | 86.1 | — | |
| JointLKCategory=KG-enhanced LM2023.12 | 85.6 | — | |
| GreaseLMCategory=KG-enhanced LM2023.12 | 84.8 | — | |
| HamQACategory=KG-enhanced LM2023.12 | 84.6 | — | |
| T5*additional_input=science facts, number of parameters=3B2021.10 | 83.2 | — | |
| T5*Additional science facts input=true, Model size (parameters)=3B2021.04 | 83.2 | — | |
| AristoRoBERTa + QA-GNNadditional_input=science facts2021.10 | 82.8 | — | |
| AristoRoBERTa + QA-GNNAdditional science facts input=true, Model size (parameters)=~360M2021.04 | 82.8 | — | |
| QA-GNNCategory=KG-enhanced LM2023.12 | 82.8 | — | |
| GrapeQACategory=KG-enhanced LM2023.12 | 82.4 | — | |
| ALBERT + KBadditional_input=science facts2021.10 | 81 | — | |
| Albert + KBAdditional science facts input=true2021.04 | 81 | — | |
| AristoRoBERTa + MHGRNadditional_input=science facts2021.10 | 80.6 | — | |
| AristoROBERTa + MHGRNAdditional science facts input=true2021.04 | 80.6 | — | |
| MHGRNCategory=KG-enhanced LM2023.12 | 80.6 | — | |
| AristoRoBERTa + PGadditional_input=science facts2021.10 | 80.2 | — | |
| AristoROBERTa + PGAdditional science facts input=true2021.04 | 80.2 | — | |
| KF + SIRadditional_input=science facts2021.10 | 80 | — | |
| KF + SIRAdditional science facts input=true2021.04 | 80 | — | |
| SFTtraining=fine-tuned on 3 datasets plus additional OBQA training data2025.02 | 77.92 | -2.74 | |
| AristoRoBERTaadditional_input=science facts2021.10 | 77.8 | — | |
| AristoRoBERTaAdditional science facts input=true2021.04 | 77.8 | — | |
| MAT-STEERmethod_type=Multi-Attribute Targeted Steering2025.02 | 77.46 | -0.32 | |
| Ours2023.05 | 74.93 | — | |
| LITOmethod_type=ITI baseline2025.02 | 74.57 | -1.12 | |
| ICLmethod_type=In-Context Learning2025.02 | 73.4 | -0.14 | |
| Llama3Category=LLM + Zero-shot, Parameters=8b2023.12 | 73 | — | |
| Careful Selectionadditional_input=science facts2021.10 | 72 | — | |
| Careful SelectionAdditional science facts input=true2021.04 | 72 | — | |
| SALKG-CoarseBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 71.29 | — | |
| PathGenBackbone=RoBERTa2021.04 | 71.2 | — | |
| SALKG-HybridBackbone=RoBERTa, Knowledge Graph Component=MHGRN, Explanation Type=Hybrid2021.04 | 70.75 | — | |
| SALKG-HybridBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 70.47 | — | |
| GSC2023.05 | 70.33 | — | |
| SALKG-HybridBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 70.2 | — | |
| SALKG-CoarseBackbone=RoBERTa, KG-augmented model architecture=PathGen2021.04 | 70.02 | — | |
| SALKG-HybridBackbone=RoBERTa, Knowledge Graph Component=PathGen, Explanation Type=Hybrid2021.04 | 70 | — | |
| SALKG-CoarseBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 69.93 | — | |
| SALKG-CoarseBackbone=RoBERTa, Knowledge Graph Component=MHGRN, Explanation Type=Coarse2021.04 | 69.85 | — | |
| SALKG-HybridBackbone=RoBERTa, KG-augmented model architecture=PathGen2021.04 | 69.8 | — | |
| SALKG-CoarseBackbone=RoBERTa, Knowledge Graph Component=PathGen, Explanation Type=Coarse2021.04 | 69.7 | — | |
| No-KG + KGBackbone=RoBERTa, KG-augmented model architecture=PathGen2021.04 | 69.67 | — | |
| RANDOM-HybridBackbone=RoBERTa, KG-augmented model architecture=PathGen2021.04 | 69.53 | — | |
| No-KG + KGBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 69.4 | — | |
| RANDOM-HybridBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 69.27 | — | |
| HEURISTIC-CoarseBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 69.24 | — | |
| SAFE2023.05 | 69.2 | — | |
| KGBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 68.87 | — | |
| UnifiedQA_T5-FTModel Size=LARGE, Protocol=Fine-Tuning2022.04 | 68.8 | — | |
| No-KGBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 68.73 | — | |
| No-KGBackbone=RoBERTa, KG-augmented model architecture=PathGen2021.04 | 68.73 | — | |
| No-KGBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 68.73 | — | |
| HEURISTIC-HybridBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 68.6 | — | |
| No-KG + KGBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 68.53 | — | |
| HEURISTIC-HybridBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 68.47 | — | |
| UnifiedQA_T5*Model Size=LARGE, Reference=Khashabi et al. (2020)2022.04 | 68.4 | — | |
| UnifiedQA_T5Model Size=LARGE2022.04 | 68.4 | — | |
| KGBackbone=RoBERTa, KG-augmented model architecture=PathGen2021.04 | 68.4 | — | |
| HEURISTIC-HybridBackbone=RoBERTa, KG-augmented model architecture=PathGen2021.04 | 68.4 | — | |
| RANDOM-HybridBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 68.33 | — | |
| RANDOM-CoarseBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 68.11 | — | |
| QA-GNN2023.05 | 67.8 | — | |
| RANDOM-CoarseBackbone=RoBERTa, KG-augmented model architecture=PathGen2021.04 | 67.18 | — | |
| GreaseLM2023.05 | 66.99 | — | |
| GenMC_T5Model Size=LARGE2022.04 | 66.87 | — | |
| MHGRNBackbone=RoBERTa2021.04 | 66.85 | — | |
| MHGRN2023.05 | 66.85 | — | |
| KGBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 66.8 | — | |
| RN + Link PredictionBackbone=RoBERTa2021.04 | 66.3 | — | |
| HEURISTIC-CoarseBackbone=RoBERTa, KG-augmented model architecture=PathGen2021.04 | 65.58 | — | |
| GPT-3Model Size=175B, Evaluation Protocol=Few-shot, Shots=1002021.12 | 65.4 | — | |
| RNBackbone=RoBERTa2021.04 | 65.2 | — | |
| RN2023.05 | 65.2 | — | |
| RANDOM-CoarseBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 65.02 | — | |
| SALKG-FineBackbone=RoBERTa, KG-augmented model architecture=MHGRN2021.04 | 64.82 | — | |
| RoBERTa-largeKnowledge Graph=without2023.05 | 64.8 | — | |
| RoBerta-largeCategory=LM + Fine tuning2023.12 | 64.8 | — | |
| GconAttnBackbone=RoBERTa2021.04 | 64.75 | — | |
| GconAttn2023.05 | 64.75 | — | |
| SALKG-FineBackbone=RoBERTa, Knowledge Graph Component=MHGRN, Explanation Type=Fine2021.04 | 64.65 | — | |
| RoBERTaModel Size=LARGE2022.04 | 64.47 | — | |
| HEURISTIC-CoarseBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 64.29 | — | |
| GLaMModel Size=64B/64E, Evaluation Protocol=Few-shot, Shots=322021.12 | 63 | — | |
| RGCNBackbone=RoBERTa2021.04 | 62.45 | — | |
| RGCN2023.05 | 62.45 | — | |
| SALKG-FineBackbone=RoBERTa, KG-augmented model architecture=RN2021.04 | 62.29 | — | |
| GenMC_T5Model Size=BASE2022.04 | 61.67 | — | |
| Bert-largeCategory=LM + Fine tuning2023.12 | 60.2 | — | |
| GPT-3.5Category=LLM + Zero-shot, Implementation=gpt-3.5-turbo2023.12 | 60 | — | |
| UnifiedQA_T5Model Size=BASE2022.04 | 59.6 | — | |
| GPT-3Model Size=175B, Evaluation Protocol=One-shot2021.12 | 58.8 | — | |
| UnifiedQA_T5-FTModel Size=BASE, Protocol=Fine-Tuning2022.04 | 58.47 | — | |
| GPT-3Model Size=175B, Evaluation Protocol=Zero-shot2021.12 | 57.6 | — |