Common Sense Reasoning on WSC273
91.58AccuracyOK-Transformer
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| OK-TransformerLM=RoBERTa, Avg.=84.75, Speed-up=1.0x2023.05 | 91.58 | — | — | — | — | — | |
| Frozen knowledgeLM=RoBERTa, Avg.=80.01, Speed-up=1.5x2023.05 | 90.48 | — | — | — | — | — | |
| Batch PartitioningLM=RoBERTa, Avg.=85.14, Speed-up=1.4x2023.05 | 90.48 | — | — | — | — | — | |
| RoBERTaLM=RoBERTa, Avg.=83.952023.05 | 90.1 | — | — | — | — | — | |
| BERT_WIKI_WSCRPre-training=MaskedWiki, Fine-tuned on WSCR=true2019.05 | 72.5 | 72 | 75.7 | 73.2 | 71 | 55 | |
| BERT_WSCRFine-tuned on WSCR=true2019.05 | 71.4 | 69.9 | 81.1 | 69.5 | 70.2 | 55 | |
| BERT_WIKI_WSCR_pairsPre-training=MaskedWiki (pairs only), Fine-tuned on WSCR=true2019.05 | 70.3 | 69.5 | 75.7 | 71.8 | 71 | 56.5 | |
| GPT_WSCRFine-tuned on WSCR=true2019.05 | 67.4 | 65.3 | 81.1 | 66.4 | 58 | 64.1 | |
| OK-TransformerLM=BERT, Avg.=66.56, Speed-up=1.0x2023.05 | 67.4 | — | — | — | — | — | |
| Batch PartitioningLM=BERT, Avg.=67.40, Speed-up=1.4x2023.05 | 66.67 | — | — | — | — | — | |
| BERT_WIKI_WSCR_no_pairsPre-training=MaskedWiki (no pairs), Fine-tuned on WSCR=true2019.05 | 66.3 | 66.9 | 62.2 | 67.2 | 64.1 | 51.1 | |
| BERTLM=BERT, Avg.=65.442023.05 | 66.3 | — | — | — | — | — | |
| Frozen knowledgeLM=BERT, Avg.=65.59, Speed-up=1.4x2023.05 | 65.93 | — | — | — | — | — | |
| LM ensemble2019.05 | 63.7 | 60.6 | 83.8 | 63.4 | 53.4 | 44.3 | |
| BERT-base_WSCRFine-tuned on WSCR=true2019.05 | 62.3 | 60.6 | 73 | 61.1 | 63.4 | 44.3 | |
| BERT_WIKIPre-training=MaskedWiki, Fine-tuned on WSCR=false2019.05 | 61.9 | 59.7 | 75.7 | 57.3 | 60.3 | 38.9 | |
| BERTFine-tuned on WSCR=false2019.05 | 61.9 | 60.2 | 73 | 59.5 | 57.3 | 45.8 | |
| Knowledge Hunter2019.05 | 57.1 | 58.3 | 50 | 58.8 | 58.8 | 90.1 | |
| BERT-baseFine-tuned on WSCR=false2019.05 | 56.4 | 55.1 | 64.9 | 52.7 | 56.5 | 44.3 | |
| GPTFine-tuned on WSCR=false2019.05 | 55.3 | 52.5 | 73 | 59.5 | 51.9 | 46.6 | |
| FairSeqNumber of Parameters=6.7B, Evaluation Protocol=5-shot2022.04 | 50 | — | — | — | — | — | |
| FairSeqNumber of Parameters=13B, Evaluation Protocol=5-shot2022.04 | 40.4 | — | — | — | — | — | |
| FairSeqNumber of Parameters=125M, Evaluation Protocol=5-shot2022.04 | 36.5 | — | — | — | — | — | |
| FairSeqNumber of Parameters=355M, Evaluation Protocol=5-shot2022.04 | 36.5 | — | — | — | — | — | |
| FairSeqNumber of Parameters=1.3B, Evaluation Protocol=5-shot2022.04 | 36.5 | — | — | — | — | — | |
| FairSeqNumber of Parameters=2.7B, Evaluation Protocol=5-shot2022.04 | 35.6 | — | — | — | — | — |