List your company
ResearchFormerly WizWand

Commonsense Question Answering on CSQA (test)

0.953AccuracyHuman
0.551560.655780.760.86422Jun 6, 2019Apr 7, 2020Feb 8, 2021Dec 11, 2021Oct 14, 2022Aug 16, 2023Jun 18, 2024
Updated 8mo ago

Evaluation results

MethodLinks
Human2019.060.953——
GPT-4o2024.060.92541.65—
Mistral-Nemo 12B2024.060.90172.06—
CPACEType=Beyond Human Level, Model setting=Ensemble2023.050.898——
KEARType=Beyond Human Level, Model setting=Ensemble2023.050.894——
Human PerformanceType=Beyond Human Level, Model setting=Ensemble2023.050.889——
Gemma-2 9B2024.060.88861.94—
CPACEType=Beyond Human Level, Model setting=Single2023.050.874——
SCORE (Oracle Verifier)Base LM=LLaMA-2-13B-chat, Verifier=oracle, Refiner=SCORE (fine-tuned)2024.040.862—100
Claude 3.5 Sonnet2024.060.86152.01—
KEARType=Beyond Human Level, Model setting=Single2023.050.861——
SCORE (Oracle Verifier)Base LM=Gemma-7B-it, Verifier=oracle, Refiner=SCORE (fine-tuned)2024.040.854—100
Llama-3 8B2024.060.84762.43—
Gemma 7B2024.060.83782.5—
ALBERT + DESC-KCRType=PLM + Symbolic Knowledge Retrieval, Model setting=Ensemble2023.050.833——
Mistral 7B2024.060.82062.49—
ALBERT + KDType=PLM + Symbolic Knowledge Retrieval, Model setting=Ensemble2023.050.809——
ALBERT + DESC-KCRType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.050.807——
ALBERT + KDType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.050.803——
ALBERT + HGNType=PLM + Generated Knowledge, Model setting=Ensemble2023.050.8——
DeBERTaType=Pre-trained Language Model Only, Model setting=Ensemble, number of parameters=1.5B2023.050.796——
ALBERT + KCRType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.050.795——
UnifiedQAType=Pre-trained Language Model Only, Model setting=Single2023.050.791——
ALBERT + HeadhunterType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.050.784——
ALBERT + PathGeneratorType=PLM + Generated Knowledge, Model setting=Ensemble2023.050.782——
T5Type=Pre-trained Language Model Only, Model setting=Single2023.050.781——
ALBERT + HGNType=PLM + Generated Knowledge, Model setting=Single2023.050.773——
CoTAMEvaluation Protocol=ICL, Number of shots (K)=32023.070.773——
FlipDA++Evaluation Protocol=ICL, Number of shots (K)=32023.070.77——
TEGBERTType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.050.768——
Extra AnnotationEvaluation Protocol=ICL, Number of shots (K)=Increased2023.070.768——
COTDAEvaluation Protocol=ICL, Number of shots (K)=32023.070.767——
ALBERTType=Pre-trained Language Model Only, Model setting=Ensemble2023.050.765——
RoBERTa + MHGRNType=PLM + Symbolic Knowledge Retrieval, Model setting=Ensemble2023.050.765——
LLM Pseudo LabelEvaluation Protocol=ICL, Number of shots (K)=32023.070.765——
BaseEvaluation Protocol=ICL, Number of shots (K)=32023.070.763——
QA-GNNType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.050.761——
ALBERT + PathGeneratorType=PLM + Generated Knowledge, Model setting=Single2023.050.756——
RoBERTa + MHGRNType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.050.754——
SCORE (GPT-4 Verifier)Base LM=Gemma-7B-it, Verifier=gpt-4, Refiner=SCORE (fine-tuned)2024.040.75—82.7
ICL_sameselection_strategy=ICL, example_type=same2024.040.746——
ICL_bioselection_strategy=ICL, example_type=bio2024.040.741——
SALKG-CoarseBackbone=RoBERTa, Graph Encoder=MHGRN2021.040.7401——
SALKG-HybridBackbone=RoBERTa, Graph Encoder=MHGRN2021.040.7387——
UnifiedQA_T5-FTModel Size=LARGE, Protocol=Fine-Tuning2022.040.736——
ALBERTType=Pre-trained Language Model Only, Model setting=Single2023.050.735——
No ExampleEvaluation Protocol=ICL, Number of shots (K)=02023.070.735——
N/Aselection_strategy=none2024.040.734——
SALKG-HybridBackbone=RoBERTa, Graph Encoder=PathGen2021.040.7303——
Random_diffselection_strategy=random, example_type=different2024.040.729——
SALKG-CoarseBackbone=RoBERTa, Graph Encoder=PathGen2021.040.7276——
PathGenBackbone=RoBERTa2021.040.7268——
SALKG-FineBackbone=RoBERTa, Graph Encoder=MHGRN2021.040.7268——
GenMC_T5Model Size=LARGE2022.040.7267——
G-DAUG-ComboSelection Strategy=Combo2020.040.726——
GenMCType=PLM + Generated Knowledge, Model setting=Single2023.050.726——
Random_bioselection_strategy=random, example_type=bio2024.040.726——
RoBERTaType=Pre-trained Language Model Only, Model setting=Ensemble2023.050.725——
SCORE (GPT-4 Verifier)Base LM=LLaMA-2-13B-chat, Verifier=gpt-4, Refiner=SCORE (fine-tuned)2024.040.724—80.3
G-DAUG-DiversitySelection Strategy=Diversity2020.040.723——
ROBERTA (reported)Source=Public leaderboards2020.040.721——
G-DAUG-InfluenceSelection Strategy=Influence2020.040.721——
RoBERTaType=Pre-trained Language Model Only, Model setting=Single2023.050.721——
RoBERTaModel Size=LARGE2022.040.7188——
G-DAUG-RandSelection Strategy=Random2020.040.718——
ROBERTA (ours)Setup=Paper setup2020.040.716——
SALKG-FineBackbone=RoBERTa, Graph Encoder=PathGen2021.040.7121——
GATBackbone=RoBERTa2021.040.712——
Random_sameselection_strategy=random, example_type=same2024.040.712——
GNBackbone=RoBERTa2021.040.7112——
MHGRNBackbone=RoBERTa2021.040.7111——
Relevantselection_strategy=relevant examples2024.040.708——
Coherence Boosting (GPT-3 175B)alpha=-0.682021.100.7043——
BACKTRANSLATION2020.040.702——
RNBackbone=RoBERTa2021.040.7008——
GconAttnBackbone=RoBERTa2021.040.6988——
Few-shot promptingBase LM=LLaMA-2-13B-chat2024.040.697——
RN + Link PredictionBackbone=RoBERTa2021.040.6933——
Gemma 7BFramework=Adaptive HPF2024.060.6855——
RGCNBackbone=RoBERTa2021.040.6841——
Phi-3 3.8BFramework=Adaptive HPF2024.060.6802——
GPT-3 175Balpha=-12021.100.6798——
Few-shot promptingBase LM=Gemma-7B-it2024.040.672——
CAGE-reasoningtraining_mode=reasoning, inference_mode=reasoning2019.060.647——
GenMC_T5Model Size=BASE2022.040.6345——
BERT + OMCSType=PLM + Symbolic Knowledge Retrieval, Model setting=Single2023.050.625——
UnifiedQA_T5-FTModel Size=BASE, Protocol=Fine-Tuning2022.040.6235——
BERT_CS_largeSize=large2019.080.622——
UnifiedQA_T5Model Size=LARGE2022.040.6134——
GPT-3 175Balpha=02021.100.611——
UnifiedQA_T5*Model Size=LARGE, Reference=Khashabi et al. (2020)2022.040.609——
CoS-E-open-endedtraining_mode=using open-ended explanations2019.060.602——
Mistral 7BFramework=Adaptive HPF2024.060.6011——
Llama-3 8BFramework=Adaptive HPF2024.060.5945——
ALBERTModel Size=LARGE2022.040.5932——
UnifiedQA_T5*Model Size=BASE, Reference=Khashabi et al. (2020)2022.040.59——
RoBERTaModel Size=BASE2022.040.5891——
CoS-E-open-ended2019.060.582——
CoS-E2019.080.582——
BERT baseline2019.060.567——