Question Answering on QASC
89.6ScoreUNIFIEDQA
Evaluation results
| Method | Links | |
|---|---|---|
| UNIFIEDQAModel Architecture=T5, Retrieval Context (IR)=Yes2020.05 | 89.6 | |
| T5Model Architecture=T5, Retrieval Context (IR)=Yes2020.05 | 88.5 | |
| KF+SIR+2Stepseen_dataset_during_training=true2020.05 | 85.2 | |
| RoBERTaModel Architecture=RoBERTa, Retrieval Context (IR)=No2020.05 | 85.2 | |
| KF+SIR + 2StepModel Architecture=RoBERTa, Retrieval Context (IR)=Yes2020.05 | 85.2 | |
| UNIFIEDQAModel Architecture=T5, Retrieval Context (IR)=No2020.05 | 78.5 | |
| UnifiedQA-BARTModel Architecture=BART, Retrieval Context (IR)=Yes2020.05 | 78.2 | |
| T5Model Architecture=T5, Retrieval Context (IR)=No2020.05 | 77 | |
| BART-largeModel Architecture=BART-large, Retrieval Context (IR)=Yes2020.05 | 75.3 | |
| UnifiedQAseen_dataset_during_training=false, training_format=multi-format2020.05 | 68.5 | |
| UnifiedQA [MC]seen_dataset_during_training=false, training_format=multiple choice2020.05 | 67.9 | |
| UnifiedQA [EX]seen_dataset_during_training=false, training_format=extractive2020.05 | 55.3 | |
| UnifiedQA [AB]seen_dataset_during_training=false, training_format=abstractive2020.05 | 54.1 | |
| UnifiedQA-BARTModel Architecture=BART, Retrieval Context (IR)=No2020.05 | 53.2 | |
| BART-largeModel Architecture=BART-large, Retrieval Context (IR)=No2020.05 | 50 | |
| Z-LaVI (OPT-30B)evaluation_protocol=zero-shot, # Param.=30B, variant=Z-LaVI2022.10 | 42.1 | |
| Z-LaVI (GPT-J-6B)evaluation_protocol=zero-shot, # Param.=6B, variant=Z-LaVI2022.10 | 42 | |
| OPT-30Bevaluation_protocol=zero-shot, # Param.=30B, variant=Original2022.10 | 39.7 | |
| Z-LaVI (GPT-Neo-2.7B)evaluation_protocol=zero-shot, # Param.=2.7B, variant=Z-LaVI2022.10 | 39.6 | |
| Z-LaVI (SBERT)evaluation_protocol=zero-shot, # Param.=110M, labeled_pretraining=true, variant=Z-LaVI2022.10 | 38.6 | |
| Z-LaVI (GPT-Neo-1.3B)evaluation_protocol=zero-shot, # Param.=1.3B, variant=Z-LaVI2022.10 | 37.4 | |
| SBERTevaluation_protocol=zero-shot, # Param.=110M, labeled_pretraining=true, variant=Original2022.10 | 36.7 | |
| GPT-J-6Bevaluation_protocol=zero-shot, # Param.=6B, variant=Original2022.10 | 36.3 | |
| Z-LaVI (SimCSE)evaluation_protocol=zero-shot, # Param.=355M, variant=Z-LaVI2022.10 | 33.2 | |
| Z-LaVI w/o LMevaluation_protocol=zero-shot, # Param.=150M2022.10 | 32.7 | |
| SimCSEevaluation_protocol=zero-shot, # Param.=355M, variant=Original2022.10 | 30.8 | |
| GPT-Neo-2.7Bevaluation_protocol=zero-shot, # Param.=2.7B, variant=Original2022.10 | 29.6 | |
| GPT-Neo-1.3Bevaluation_protocol=zero-shot, # Param.=1.3B, variant=Original2022.10 | 29.3 | |
| Z-LaVI (BART-L-mnli)evaluation_protocol=zero-shot, # Param.=400 M, labeled_pretraining=true, variant=Z-LaVI2022.10 | 27.3 | |
| Z-LaVI (ROBERTa-L-mnli)evaluation_protocol=zero-shot, # Param.=355 M, labeled_pretraining=true, variant=Z-LaVI2022.10 | 27.2 | |
| SMLMevaluation_protocol=zero-shot, # Param.=355 M2022.10 | 26.6 | |
| BART-L-mnlievaluation_protocol=zero-shot, # Param.=400 M, labeled_pretraining=true, variant=Original2022.10 | 21.7 | |
| ROBERTa-L-mnlievaluation_protocol=zero-shot, # Param.=355 M, labeled_pretraining=true, variant=Original2022.10 | 19.3 | |
| IR Solverevaluation_protocol=zero-shot2022.10 | 18.6 | |
| UnifiedQA [YN]seen_dataset_during_training=false, training_format=yes/no2020.05 | 14.8 | |
| Randomevaluation_protocol=zero-shot2022.10 | 12.5 |