Natural Language Inference on ANLI
73.1ANLI R1 AccuracyPaLM-2 L
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| PaLM-2 Lprotocol=1-shot2023.11 | 73.1 | — | 63.4 | — | 67.1 | — | |
| Falcon-180Bprotocol=1-shot2023.11 | 60.5 | — | 55.5 | — | 56.8 | — | |
| PaLM-2 Mprotocol=1-shot2023.11 | 58.1 | — | 49.5 | — | 54.5 | — | |
| PaLM-2 Sprotocol=1-shot2023.11 | 53.1 | — | 48.8 | — | 53.2 | — | |
| PaLMprotocol=1-shot2023.11 | 52.6 | — | 48.7 | — | 52.3 | — | |
| KCRegArchitecture=T5 [57], Parameters=~220M2024.11 | 48.2 | — | 45 | — | 44.3 | — | |
| KCRegArchitecture=GPT2 [56], Parameters=~1.5B2024.11 | 47.1 | — | 48.1 | — | 44.7 | — | |
| BaseArchitecture=T5 [57], Parameters=~220M2024.11 | 46.1 | — | 44.7 | — | 46 | — | |
| KCRegArchitecture=BERT [16], Parameters=~110M2024.11 | 45.6 | — | 46.9 | — | 45.3 | — | |
| ALUMArchitecture=BERT [16], Parameters=~110M2024.11 | 45.2 | — | 46.7 | — | 46.3 | — | |
| T0-11BParams=11B, Evaluation=Zero-shot2022.10 | 44.7 | 3.6 | 39.4 | 2.2 | 42.4 | 3 | |
| BaseArchitecture=BERT [16], Parameters=~110M2024.11 | 44.5 | — | 45.6 | — | 33.8 | — | |
| ALUMArchitecture=T5 [57], Parameters=~220M2024.11 | 44.5 | — | 44.8 | — | 44.4 | — | |
| ALUMArchitecture=GPT2 [56], Parameters=~1.5B2024.11 | 43.8 | — | 45.2 | — | 44.6 | — | |
| BaseArchitecture=GPT2 [56], Parameters=~1.5B2024.11 | 42.7 | — | 44.9 | — | 43.4 | — | |
| KiC-LargeParams=0.77B, Evaluation=Zero-shot2022.10 | 36.3 | 1.8 | 35 | 1.4 | 37.6 | 2.5 | |
| GPT-JParams=6B, Evaluation=Zero-shot2022.10 | 34 | 1.2 | 33.6 | 0.7 | 33.6 | 0.7 | |
| BERTParams=0.34B, Evaluation=Zero-shot2022.10 | 33.8 | 1.4 | 33.8 | 0.9 | 33.2 | 0.6 | |
| T0-3BParams=3B, Evaluation=Zero-shot2022.10 | 33.7 | 0.9 | 33.4 | 1.2 | 33.3 | 0.4 | |
| GPT-NeoParams=2.7B, Evaluation=Zero-shot2022.10 | 33.5 | 1.1 | 33.7 | 0.9 | 33.4 | 0.5 | |
| RoBERTaParams=0.35B, Evaluation=Zero-shot2022.10 | 33.4 | 0.9 | 33.4 | 0.6 | 33.3 | 0.5 | |
| GPT-NeoXParams=20B, Evaluation=Zero-shot2022.10 | 33.4 | 1.2 | 33.4 | 1 | 33.6 | 0.7 | |
| OPTParams=30B, Evaluation=Zero-shot2022.10 | 33.3 | 0.2 | 33.3 | 0.1 | 33.4 | 0.1 | |
| KiC-SmallParams=0.06B, Evaluation=Zero-shot2022.10 | 33.3 | 1 | 33.3 | 0.9 | 33.6 | 0.6 | |
| T0-BaseParams=0.22B, Evaluation=Zero-shot2022.10 | 32.2 | 1.4 | 33 | 1 | 34.2 | 0.7 | |
| T0-LargeParams=0.77B, Evaluation=Zero-shot2022.10 | 30.5 | 2 | 32.7 | 0.6 | 33.8 | 0.8 | |
| KiC-BaseParams=0.22B, Evaluation=Zero-shot2022.10 | 28.4 | 2.4 | 30.9 | 1.7 | 32.8 | 1.1 |