Commonsense Reasoning on CSQA (test)
89.4AccuracyKEAR
Evaluation results
| Method | Links | ||
|---|---|---|---|
| KEAREvaluation Protocol=Ensemble2021.12 | 89.4 | — | |
| HumanEvaluation Protocol=Ensemble2021.12 | 88.9 | — | |
| Self-ConsistencyModel=GPT-4, Scenario=Third scenario (type and format known)2023.11 | 88.1 | — | |
| Self-AgreementModel=GPT-4, Scenario=Third scenario (type and format known)2023.11 | 87.9 | — | |
| Few-Shot CoTModel=GPT-4, Scenario=Third scenario (type and format known)2023.11 | 86.9 | — | |
| KEAREvaluation Protocol=Single2021.12 | 86.1 | — | |
| SGEModel=GPT-4, Tool=Code Interpreter2024.05 | 85.68 | — | |
| Decomp PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 84.88 | — | |
| CoT PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 84.15 | — | |
| Refine PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 83.5 | — | |
| DEKCOREvaluation Protocol=Ensemble2021.12 | 83.3 | — | |
| Few-shot*Backbone=Qwen2.5-7B, Few-shot settings=5-shot ID2025.09 | 82.2 | — | |
| DRR2025.03 | 82.1 | — | |
| ICRBackbone=Qwen2.5-7B2025.09 | 82 | — | |
| ALBERT + SFREvaluation Protocol=Ensemble2021.12 | 81.8 | — | |
| M²IVBackbone=Qwen2.5-7B2025.09 | 81.8 | — | |
| I2CLBackbone=Qwen2.5-7B2025.09 | 81.6 | — | |
| LIVEBackbone=Qwen2.5-7B2025.09 | 81.6 | — | |
| IO PromptingModel=GPT-4, Tool=Code Interpreter2024.05 | 81.14 | — | |
| ALBERT + KDEvaluation Protocol=Ensemble2021.12 | 80.9 | — | |
| DEKCOREvaluation Protocol=Single2021.12 | 80.7 | — | |
| Zero-shotBackbone=Qwen2.5-7B2025.09 | 80.4 | — | |
| ALBERT + KDEvaluation Protocol=Single2021.12 | 80.3 | — | |
| ALBERT + HGNEvaluation Protocol=Ensemble2021.12 | 80 | — | |
| Entro-duction2025.03 | 79.6 | 7.1 | |
| ALBERT+KCREvaluation Protocol=Single2021.12 | 79.5 | — | |
| UnifiedQAEvaluation Protocol=Single2021.12 | 79.1 | — | |
| CoT-SC@maj64majority_vote=642025.03 | 78.7 | 15 | |
| CoT-SC@maj8majority_vote=82025.03 | 78.2 | 15 | |
| T5Evaluation Protocol=Single2021.12 | 78.1 | — | |
| Self-ConsistencyModel=Llama-2-70B-Chat, Scenario=Third scenario (type and format known)2023.11 | 78 | — | |
| Self-AgreementModel=Llama-2-70B-Chat, Scenario=Third scenario (type and format known)2023.11 | 77.7 | — | |
| ALBERT + HGNEvaluation Protocol=Single2021.12 | 77.3 | — | |
| Few-Shot CoTModel=Llama-2-70B-Chat, Scenario=Third scenario (type and format known)2023.11 | 77.1 | — | |
| ALBERTEvaluation Protocol=Ensemble2021.12 | 76.5 | — | |
| ROBERTa+MHGRNEvaluation Protocol=Ensemble2021.12 | 76.5 | — | |
| CoT2025.03 | 75.6 | 5 | |
| ROBERTa+MHGRNEvaluation Protocol=Single2021.12 | 75.4 | — | |
| ROBERTa+KEDGNEvaluation Protocol=Ensemble2021.12 | 74.4 | — | |
| SALKG-HybridLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 74.13 | — | |
| Complex CoT2025.03 | 73.9 | 5 | |
| SALKG-CoarseLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 73.64 | — | |
| ToT2025.03 | 73.5 | 121 | |
| SALKG-HybridLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 73.37 | — | |
| KGLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 73.14 | — | |
| SALKG-CoarseLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 73.11 | — | |
| RANDOM-FineLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 73.09 | — | |
| SALKG-CoarseLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 73.07 | — | |
| SALKG-HybridLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 72.67 | — | |
| KGLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 72.58 | — | |
| HEURISTIC-HybridLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 72.58 | — | |
| HEURISTIC-CoarseLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 72.57 | — | |
| ROBERTaEvaluation Protocol=Ensemble2021.12 | 72.5 | — | |
| No-KG + KGLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 72.44 | — | |
| No-KG + KGLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 72.15 | — | |
| HEURISTIC-CoarseLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 72.15 | — | |
| SALKG-FineLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 72.12 | — | |
| ROBERTaEvaluation Protocol=Single2021.12 | 72.1 | — | |
| RANDOM-HybridLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 71.93 | — | |
| SALKG-FineLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 71.64 | — | |
| HEURISTIC-FineLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 71.5 | — | |
| KGLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 71.37 | — | |
| RANDOM-HybridLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 71.35 | — | |
| HEURISTIC-HybridLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 71.33 | — | |
| RANDOM-FineLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 71.26 | — | |
| HEURISTIC-CoarseLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 71.18 | — | |
| No-KG + KGLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 71.15 | — | |
| RANDOM-CoarseLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 71.15 | — | |
| HEURISTIC-FineLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 71.11 | — | |
| RANDOM-CoarseLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 71.06 | — | |
| RANDOM-HybridLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 70.12 | — | |
| Self-talk2025.03 | 70 | — | |
| SALKG-FineLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 70 | — | |
| No-KGLanguage Model=RoBERTa, KG Reasoning Model=MHGRN2021.04 | 69.65 | — | |
| No-KGLanguage Model=RoBERTa, KG Reasoning Model=PathGen2021.04 | 69.65 | — | |
| No-KGLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 69.65 | — | |
| RANDOM-FineLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 69.08 | — | |
| RANDOM-CoarseLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 69.06 | — | |
| HEURISTIC-FineLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 65.08 | — | |
| HEURISTIC-HybridLanguage Model=RoBERTa, KG Reasoning Model=RN2021.04 | 65.07 | — | |
| BERT+OMCSEvaluation Protocol=Single2021.12 | 62.5 | — | |
| SALKG-HybridLanguage Model=BERT, KG Reasoning Model=PathGen2021.04 | 59.87 | — | |
| SALKG-HybridLanguage Model=BERT, KG Reasoning Model=RN2021.04 | 58.78 | — | |
| SALKG-HybridLanguage Model=BERT, KG Reasoning Model=MHGRN2021.04 | 58.7 | — | |
| Few-shot*Backbone=Llama2-7B, Few-shot settings=5-shot ID2025.09 | 58 | — | |
| SALKG-CoarseLanguage Model=BERT, KG Reasoning Model=MHGRN2021.04 | 57.98 | — | |
| SALKG-CoarseLanguage Model=BERT, KG Reasoning Model=PathGen2021.04 | 57.75 | — | |
| SALKG-CoarseLanguage Model=BERT, KG Reasoning Model=RN2021.04 | 57.5 | — | |
| KGLanguage Model=BERT, KG Reasoning Model=MHGRN2021.04 | 57.48 | — | |
| No-KG + KGLanguage Model=BERT, KG Reasoning Model=PathGen2021.04 | 57.29 | — | |
| HEURISTIC-CoarseLanguage Model=BERT, KG Reasoning Model=PathGen2021.04 | 56.92 | — | |
| HEURISTIC-HybridLanguage Model=BERT, KG Reasoning Model=PathGen2021.04 | 56.83 | — | |
| KGLanguage Model=BERT, KG Reasoning Model=PathGen2021.04 | 56.54 | — | |
| KGLanguage Model=BERT, KG Reasoning Model=RN2021.04 | 56.46 | — | |
| HEURISTIC-CoarseLanguage Model=BERT, KG Reasoning Model=RN2021.04 | 56.42 | — | |
| HEURISTIC-HybridLanguage Model=BERT, KG Reasoning Model=MHGRN2021.04 | 56.35 | — | |
| No-KG + KGLanguage Model=BERT, KG Reasoning Model=MHGRN2021.04 | 56.14 | — | |
| No-KG + KGLanguage Model=BERT, KG Reasoning Model=RN2021.04 | 55.98 | — | |
| HEURISTIC-CoarseLanguage Model=BERT, KG Reasoning Model=MHGRN2021.04 | 55.55 | — | |
| RANDOM-HybridLanguage Model=BERT, KG Reasoning Model=PathGen2021.04 | 55.24 | — |