Commonsense Question Answering on CSQA (test)
0.953AccuracyHuman
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Human2019.06 | 0.953 | — | — | |
| GPT-4o2024.06 | 0.9254 | 1.65 | — | |
| Mistral-Nemo 12B2024.06 | 0.9017 | 2.06 | — | |
| CPACEType=Beyond Human Level, Model setting=Ensemble2023.05 | 0.898 | — | — | |
| KEARType=Beyond Human Level, Model setting=Ensemble2023.05 | 0.894 | — | — | |
| Human PerformanceType=Beyond Human Level, Model setting=Ensemble2023.05 | 0.889 | — | — | |
| Gemma-2 9B2024.06 | 0.8886 | 1.94 | — | |
| CPACEType=Beyond Human Level, Model setting=Single2023.05 | 0.874 | — | — | |
| SCORE (Oracle Verifier)Base LM=LLaMA-2-13B-chat, Verifier=oracle, Refiner=SCORE (fine-tuned)2024.04 | 0.862 | — | 100 | |
| Claude 3.5 Sonnet2024.06 | 0.8615 | 2.01 | — | |
| KEARType=Beyond Human Level, Model setting=Single2023.05 | 0.861 | — | — | |
| SCORE (Oracle Verifier)Base LM=Gemma-7B-it, Verifier=oracle, Refiner=SCORE (fine-tuned)2024.04 | 0.854 | — | 100 | |
| Llama-3 8B2024.06 | 0.8476 | 2.43 | — | |
| Gemma 7B2024.06 | 0.8378 | 2.5 | — | |
| ALBERT + DESC-KCRType=PLM + Symbolic Knowledge Retrieval, Model setting=Ensemble2023.05 | 0.833 | — | — | |
| Mistral 7B2024.06 | 0.8206 | 2.49 | — | |
| ALBERT + KDType=PLM + Symbolic Knowledge Retrieval, Model setting=Ensemble2023.05 | 0.809 | — | — | |
| ALBERT + DESC-KCRType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.05 | 0.807 | — | — | |
| ALBERT + KDType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.05 | 0.803 | — | — | |
| ALBERT + HGNType=PLM + Generated Knowledge, Model setting=Ensemble2023.05 | 0.8 | — | — | |
| DeBERTaType=Pre-trained Language Model Only, Model setting=Ensemble, number of parameters=1.5B2023.05 | 0.796 | — | — | |
| ALBERT + KCRType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.05 | 0.795 | — | — | |
| UnifiedQAType=Pre-trained Language Model Only, Model setting=Single2023.05 | 0.791 | — | — | |
| ALBERT + HeadhunterType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.05 | 0.784 | — | — | |
| ALBERT + PathGeneratorType=PLM + Generated Knowledge, Model setting=Ensemble2023.05 | 0.782 | — | — | |
| T5Type=Pre-trained Language Model Only, Model setting=Single2023.05 | 0.781 | — | — | |
| ALBERT + HGNType=PLM + Generated Knowledge, Model setting=Single2023.05 | 0.773 | — | — | |
| CoTAMEvaluation Protocol=ICL, Number of shots (K)=32023.07 | 0.773 | — | — | |
| FlipDA++Evaluation Protocol=ICL, Number of shots (K)=32023.07 | 0.77 | — | — | |
| TEGBERTType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.05 | 0.768 | — | — | |
| Extra AnnotationEvaluation Protocol=ICL, Number of shots (K)=Increased2023.07 | 0.768 | — | — | |
| COTDAEvaluation Protocol=ICL, Number of shots (K)=32023.07 | 0.767 | — | — | |
| ALBERTType=Pre-trained Language Model Only, Model setting=Ensemble2023.05 | 0.765 | — | — | |
| RoBERTa + MHGRNType=PLM + Symbolic Knowledge Retrieval, Model setting=Ensemble2023.05 | 0.765 | — | — | |
| LLM Pseudo LabelEvaluation Protocol=ICL, Number of shots (K)=32023.07 | 0.765 | — | — | |
| BaseEvaluation Protocol=ICL, Number of shots (K)=32023.07 | 0.763 | — | — | |
| QA-GNNType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.05 | 0.761 | — | — | |
| ALBERT + PathGeneratorType=PLM + Generated Knowledge, Model setting=Single2023.05 | 0.756 | — | — | |
| RoBERTa + MHGRNType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.05 | 0.754 | — | — | |
| SCORE (GPT-4 Verifier)Base LM=Gemma-7B-it, Verifier=gpt-4, Refiner=SCORE (fine-tuned)2024.04 | 0.75 | — | 82.7 | |
| ICL_sameselection_strategy=ICL, example_type=same2024.04 | 0.746 | — | — | |
| ICL_bioselection_strategy=ICL, example_type=bio2024.04 | 0.741 | — | — | |
| SALKG-CoarseBackbone=RoBERTa, Graph Encoder=MHGRN2021.04 | 0.7401 | — | — | |
| SALKG-HybridBackbone=RoBERTa, Graph Encoder=MHGRN2021.04 | 0.7387 | — | — | |
| UnifiedQA_T5-FTModel Size=LARGE, Protocol=Fine-Tuning2022.04 | 0.736 | — | — | |
| ALBERTType=Pre-trained Language Model Only, Model setting=Single2023.05 | 0.735 | — | — | |
| No ExampleEvaluation Protocol=ICL, Number of shots (K)=02023.07 | 0.735 | — | — | |
| N/Aselection_strategy=none2024.04 | 0.734 | — | — | |
| SALKG-HybridBackbone=RoBERTa, Graph Encoder=PathGen2021.04 | 0.7303 | — | — | |
| Random_diffselection_strategy=random, example_type=different2024.04 | 0.729 | — | — | |
| SALKG-CoarseBackbone=RoBERTa, Graph Encoder=PathGen2021.04 | 0.7276 | — | — | |
| PathGenBackbone=RoBERTa2021.04 | 0.7268 | — | — | |
| SALKG-FineBackbone=RoBERTa, Graph Encoder=MHGRN2021.04 | 0.7268 | — | — | |
| GenMC_T5Model Size=LARGE2022.04 | 0.7267 | — | — | |
| G-DAUG-ComboSelection Strategy=Combo2020.04 | 0.726 | — | — | |
| GenMCType=PLM + Generated Knowledge, Model setting=Single2023.05 | 0.726 | — | — | |
| Random_bioselection_strategy=random, example_type=bio2024.04 | 0.726 | — | — | |
| RoBERTaType=Pre-trained Language Model Only, Model setting=Ensemble2023.05 | 0.725 | — | — | |
| SCORE (GPT-4 Verifier)Base LM=LLaMA-2-13B-chat, Verifier=gpt-4, Refiner=SCORE (fine-tuned)2024.04 | 0.724 | — | 80.3 | |
| G-DAUG-DiversitySelection Strategy=Diversity2020.04 | 0.723 | — | — | |
| ROBERTA (reported)Source=Public leaderboards2020.04 | 0.721 | — | — | |
| G-DAUG-InfluenceSelection Strategy=Influence2020.04 | 0.721 | — | — | |
| RoBERTaType=Pre-trained Language Model Only, Model setting=Single2023.05 | 0.721 | — | — | |
| RoBERTaModel Size=LARGE2022.04 | 0.7188 | — | — | |
| G-DAUG-RandSelection Strategy=Random2020.04 | 0.718 | — | — | |
| ROBERTA (ours)Setup=Paper setup2020.04 | 0.716 | — | — | |
| SALKG-FineBackbone=RoBERTa, Graph Encoder=PathGen2021.04 | 0.7121 | — | — | |
| GATBackbone=RoBERTa2021.04 | 0.712 | — | — | |
| Random_sameselection_strategy=random, example_type=same2024.04 | 0.712 | — | — | |
| GNBackbone=RoBERTa2021.04 | 0.7112 | — | — | |
| MHGRNBackbone=RoBERTa2021.04 | 0.7111 | — | — | |
| Relevantselection_strategy=relevant examples2024.04 | 0.708 | — | — | |
| Coherence Boosting (GPT-3 175B)alpha=-0.682021.10 | 0.7043 | — | — | |
| BACKTRANSLATION2020.04 | 0.702 | — | — | |
| RNBackbone=RoBERTa2021.04 | 0.7008 | — | — | |
| GconAttnBackbone=RoBERTa2021.04 | 0.6988 | — | — | |
| Few-shot promptingBase LM=LLaMA-2-13B-chat2024.04 | 0.697 | — | — | |
| RN + Link PredictionBackbone=RoBERTa2021.04 | 0.6933 | — | — | |
| Gemma 7BFramework=Adaptive HPF2024.06 | 0.6855 | — | — | |
| RGCNBackbone=RoBERTa2021.04 | 0.6841 | — | — | |
| Phi-3 3.8BFramework=Adaptive HPF2024.06 | 0.6802 | — | — | |
| GPT-3 175Balpha=-12021.10 | 0.6798 | — | — | |
| Few-shot promptingBase LM=Gemma-7B-it2024.04 | 0.672 | — | — | |
| CAGE-reasoningtraining_mode=reasoning, inference_mode=reasoning2019.06 | 0.647 | — | — | |
| GenMC_T5Model Size=BASE2022.04 | 0.6345 | — | — | |
| BERT + OMCSType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.05 | 0.625 | — | — | |
| UnifiedQA_T5-FTModel Size=BASE, Protocol=Fine-Tuning2022.04 | 0.6235 | — | — | |
| BERT_CS_largeSize=large2019.08 | 0.622 | — | — | |
| UnifiedQA_T5Model Size=LARGE2022.04 | 0.6134 | — | — | |
| GPT-3 175Balpha=02021.10 | 0.611 | — | — | |
| UnifiedQA_T5*Model Size=LARGE, Reference=Khashabi et al. (2020)2022.04 | 0.609 | — | — | |
| CoS-E-open-endedtraining_mode=using open-ended explanations2019.06 | 0.602 | — | — | |
| Mistral 7BFramework=Adaptive HPF2024.06 | 0.6011 | — | — | |
| Llama-3 8BFramework=Adaptive HPF2024.06 | 0.5945 | — | — | |
| ALBERTModel Size=LARGE2022.04 | 0.5932 | — | — | |
| UnifiedQA_T5*Model Size=BASE, Reference=Khashabi et al. (2020)2022.04 | 0.59 | — | — | |
| RoBERTaModel Size=BASE2022.04 | 0.5891 | — | — | |
| CoS-E-open-ended2019.06 | 0.582 | — | — | |
| CoS-E2019.08 | 0.582 | — | — | |
| BERT baseline2019.06 | 0.567 | — | — |