Commonsense inference modeling on ATOMIC 20 (test)
0.5071BLEU-1CANDLE Distilled
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| CANDLE DistilledBackbone=GPT2-XL (1.5B), Training Data=CANDLE instantiations, fine-tuned on ATOMIC20=true2024.01 | 0.5071 | 0.3385 | 25.55 | 20.43 | 0.3245 | 0.5191 | 69.68 | 0.7686 | |
| GPT2-XLBackbone=GPT2-XL (1.5B), Training Data=ATOMIC, fine-tuned on ATOMIC20=true2024.01 | 0.4572 | 0.2918 | 21.12 | 16.15 | 0.2997 | 0.4969 | 64.61 | 0.7609 | |
| GPT2-XLBackbone=GPT2-XL (1.5B), Training Data=ATOMIC-10X, fine-tuned on ATOMIC20=true2024.01 | 0.4538 | 0.292 | 21.09 | 16.15 | 0.3009 | 0.4986 | 65.02 | 0.7589 | |
| GPT2-XLBackbone=GPT2-XL (1.5B), Training Data=AbstractATOMIC, fine-tuned on ATOMIC20=true2024.01 | 0.453 | 0.2908 | 21 | 16.06 | 0.2998 | 0.4861 | 63.98 | 0.7556 | |
| CANDLE DistilledBackbone=LLAMA2 (7B), Training Data=CANDLE instantiations, fine-tuned on ATOMIC20=true2024.01 | 0.4386 | 0.244 | 15.12 | 10 | 0.2836 | 0.4386 | 54.25 | 0.7294 | |
| LLAMA2Backbone=LLAMA2 (7B), Training Data=AbstractATOMIC, fine-tuned on ATOMIC20=true2024.01 | 0.4278 | 0.2364 | 14.58 | 9.471 | 0.2774 | 0.4255 | 53.12 | 0.7151 | |
| GPT2-XLBackbone=GPT2-XL (1.5B), Training Data=ATOMIC202024.01 | 0.4215 | 0.2577 | 17.82 | 13.11 | 0.2982 | 0.4761 | 63.7 | 0.7039 | |
| LLAMA2Backbone=LLAMA2 (7B), Training Data=ATOMIC-10X, fine-tuned on ATOMIC20=true2024.01 | 0.4207 | 0.2308 | 14.14 | 9.198 | 0.2814 | 0.4275 | 53.69 | 0.7193 | |
| LLAMA2Backbone=LLAMA2 (7B), Training Data=ATOMIC, fine-tuned on ATOMIC20=true2024.01 | 0.4204 | 0.2301 | 14.1 | 9.125 | 0.278 | 0.429 | 53.17 | 0.7152 | |
| LLAMA2Backbone=LLAMA2 (7B), Training Data=ATOMIC202024.01 | 0.4107 | 0.2246 | 13.62 | 8.619 | 0.2774 | 0.4242 | 53.28 | 0.7177 | |
| GPT2-XLBackbone=GPT2-XL (1.5B), Training Data=ATOMIC-10X2024.01 | 0.3369 | 0.1882 | 11.71 | 7.91 | 0.1878 | 0.2569 | 19.29 | 0.6147 | |
| LLAMA2Backbone=LLAMA2 (7B), Training Data=ATOMIC-10X2024.01 | 0.3306 | 0.1765 | 9.986 | 6.078 | 0.1922 | 0.2532 | 17.8 | 0.6125 | |
| GPT2-XLBackbone=GPT2-XL (1.5B), Training Data=ATOMIC2024.01 | 0.3223 | 0.1906 | 13.27 | 10.28 | 0.1763 | 0.255 | 20.15 | 0.5839 | |
| LLAMA2Backbone=LLAMA2 (7B), Training Data=Five-shot2024.01 | 0.3122 | 0.1687 | 9.767 | 5.989 | 0.1974 | 0.2767 | 17.83 | 0.5841 | |
| LLAMA2Backbone=LLAMA2 (7B), Training Data=ATOMIC2024.01 | 0.2994 | 0.1644 | 10.03 | 6.631 | 0.1902 | 0.2575 | 18.71 | 0.5968 | |
| GPT2-XLBackbone=GPT2-XL (1.5B), Training Data=AbstractATOMIC2024.01 | 0.2946 | 0.1716 | 11.89 | 9.019 | 0.1742 | 0.243 | 19.95 | 0.5783 | |
| CANDLE DistilledBackbone=LLAMA2 (7B), Training Data=CANDLE instantiations2024.01 | 0.2893 | 0.1556 | 9.468 | 6.14 | 0.186 | 0.2537 | 17.2 | 0.6027 | |
| CANDLE DistilledBackbone=GPT2-XL (1.5B), Training Data=CANDLE instantiations2024.01 | 0.2691 | 0.1644 | 12.31 | 10.28 | 0.1766 | 0.2366 | 21.36 | 0.5715 | |
| ChatGPTBackbone=openai/gpt-3.5-turbo, Evaluation Protocol=Five-shot2024.01 | 0.2632 | 0.125 | 7.16 | 4.415 | 0.186 | 0.2465 | 8.313 | 0.5869 | |
| LLAMA2Backbone=LLAMA2 (7B), Training Data=AbstractATOMIC2024.01 | 0.2608 | 0.1327 | 7.799 | 5.018 | 0.1508 | 0.212 | 14.78 | 0.5683 | |
| LLAMA2Backbone=LLAMA2 (7B), Training Data=Zero-shot2024.01 | 0.1826 | 0.0745 | 3.594 | 1.945 | 0.159 | 0.2028 | 8.872 | 0.4823 | |
| ChatGPTBackbone=openai/gpt-3.5-turbo, Evaluation Protocol=Zero-shot2024.01 | 0.1182 | 0.0426 | 1.891 | 0.926 | 0.1387 | 0.1373 | 4.35 | 0.4928 | |
| ChatGPTBackbone=openai/gpt-3.5-turbo, Evaluation Protocol=Chain-of-thought2024.01 | 0.0991 | 0.0357 | 1.556 | 0.736 | 0.1185 | 0.1102 | 2.905 | 0.4617 | |
| GPT2-XLBackbone=GPT2-XL (1.5B), Training Data=Zero-shot2024.01 | 0.0435 | 0.016 | 0.732 | 0.293 | 0.057 | 0.0503 | 0.792 | 0.3711 |