Commonsense Question Answering on CosmosQA
94AccuracyHUMAN
Evaluation Results
| Method | Links | |
|---|---|---|
| HUMAN2021.03 | 94 | |
| UNICORN2021.03 | 91.8 | |
| QA+CCoverage=20%, Base model=RoBERTa-RACE2022.09 | 91.12 | |
| PACSBackbone=Llama 3.3 70B2026.05 | 91 | |
| T5-11BParameters=11B2021.03 | 90.3 | |
| SC-20Backbone=Llama 3.3 70B2026.05 | 90 | |
| ARGOSBackbone=Llama 3.3 70B2026.05 | 90 | |
| QACoverage=20%, Base model=RoBERTa-RACE2022.09 | 88.61 | |
| COTBackbone=Llama 3.3 70B2026.05 | 88 | |
| If-BeamBackbone=Llama 3.3 70B2026.05 | 88 | |
| ALBERT-XXLARGEModel Size=XXLarge2021.03 | 85.4 | |
| LoTBackbone=Llama 3.3 70B2026.05 | 85 | |
| ROBERTA-LARGEModel Size=Large2021.03 | 83.5 | |
| CCoverage=20%, Base model=RoBERTa-RACE2022.09 | 83.25 | |
| ORACLEBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 82.4 | |
| DRAGON2022.10 | 82.3 | |
| ORACLEBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 82.1 | |
| PACSBackbone=Llama 3-Instruct 8B2026.05 | 82 | |
| ARGOSBackbone=Llama 3-Instruct 8B2026.05 | 82 | |
| If-BeamBackbone=Llama 3-Instruct 8B2026.05 | 82 | |
| K-ADAPTER (F+L)Adapter type=facAdapter + linAdapter2020.02 | 81.83 | |
| QA+CCoverage=50%, Base model=RoBERTa-RACE2022.09 | 81.7 | |
| self-rewardingBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 81.5 | |
| RoBERTa + multitaskmultitask=true2020.02 | 81.19 | |
| K-ADAPTER (F)Adapter type=facAdapter2020.02 | 80.93 | |
| K-ADAPTER (L)Adapter type=linAdapter2020.02 | 80.76 | |
| QAGNN2022.10 | 80.7 | |
| CCoverage=50%, Base model=RoBERTa-RACE2022.09 | 80.63 | |
| GreaseLM2022.10 | 80.6 | |
| RoBERTa2020.02 | 80.59 | |
| RoBERTa2022.10 | 80.5 | |
| RFTBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 80.4 | |
| QA+E+CCoverage=50%, Base model=RoBERTa-RACE2022.09 | 80.29 | |
| SC-20Backbone=Llama 3-Instruct 8B2026.05 | 80 | |
| CoTBackbone=Llama-3.1-8B-Instruct, Setting=four-shot2026.03 | 79.5 | |
| ORACLEBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 78.5 | |
| self-rewardingBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 78.1 | |
| QA+E+CCoverage=20%, Base model=RoBERTa-RACE2022.09 | 77.55 | |
| CoTBackbone=Qwen-2.5-7B-instruct, Setting=four-shot2026.03 | 77.1 | |
| QA+ECoverage=50%, Base model=RoBERTa-RACE2022.09 | 76.94 | |
| QA+ECoverage=20%, Base model=RoBERTa-RACE2022.09 | 76.88 | |
| QACoverage=50%, Base model=RoBERTa-RACE2022.09 | 76.47 | |
| RFTBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 76 | |
| LoTBackbone=Llama 3-Instruct 8B2026.05 | 76 | |
| CoTBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 75.9 | |
| E+CCoverage=50%, Base model=RoBERTa-RACE2022.09 | 75.8 | |
| COTBackbone=Llama 3-Instruct 8B2026.05 | 75 | |
| ToT-SFTBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 74.7 | |
| CoTBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 74.2 | |
| RFTBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 74 | |
| ToT-SFTBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 73.9 | |
| ToT-SFTBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 72.4 | |
| self-rewardingBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 71.7 | |
| CoTBackbone=Mistral-7B-Instruct-v0.3, Setting=four-shot2026.03 | 71.1 | |
| ECoverage=50%, Base model=RoBERTa-RACE2022.09 | 70.64 | |
| CoTBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 70.6 | |
| E+CCoverage=20%, Base model=RoBERTa-RACE2022.09 | 69.18 | |
| BERT-FT RACE+SWAGpre-training datasets=RACE + SWAG2020.02 | 68.7 | |
| ECoverage=20%, Base model=RoBERTa-RACE2022.09 | 68.34 | |
| QA+Cmodel=RoBERTa-RACE, contradiction=true2022.09 | 64.82 | |
| QA+E+Cmodel=RoBERTa-RACE, entailment=true, contradiction=true2022.09 | 64.72 | |
| QA+Emodel=RoBERTa-RACE, entailment=true2022.09 | 64.32 | |
| QAmodel=RoBERTa-RACE2022.09 | 64.22 | |
| LLM-TreeBackbone=Llama 3-Instruct 8B2026.05 | 51 | |
| LLM-TreeBackbone=Llama 3.3 70B2026.05 | 51 | |
| OLTQAFull Config=True2023.05 | 46.16 | |
| SciQ-smallvariant=small2022.09 | 25.46 | |
| SciQ-basevariant=base2022.09 | 18.46 |