Reading Comprehension on ReCoRD
93.8AccuracyPaLM 2-L
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PaLM 2-Lprompting=1-shot2023.05 | 93.8 | — | |
| PaLM-2 Lprotocol=1-shot2023.11 | 93.8 | — | |
| Falcon-180Bprotocol=1-shot2023.11 | 93.2 | — | |
| PaLM 540BEvaluation Protocol=0-shot2022.04 | 92.9 | — | |
| PaLM 540BEvaluation Protocol=Few-shot, Number of Shots=22022.04 | 92.9 | — | |
| PaLMprompting=1-shot2023.05 | 92.8 | — | |
| PaLMprotocol=1-shot2023.11 | 92.8 | — | |
| PaLM 540BEvaluation Protocol=1-shot2022.04 | 92.8 | — | |
| PaLM 2-Mprompting=1-shot2023.05 | 92.4 | — | |
| PaLM-2 Mprotocol=1-shot2023.11 | 92.4 | — | |
| PaLM 2-Sprompting=1-shot2023.05 | 92.1 | — | |
| PaLM-2 Sprotocol=1-shot2023.11 | 92.1 | — | |
| LLaMA# of params=7B2023.04 | 91.4 | — | |
| Prior SOTAEvaluation Protocol=Few-shot, Number of Shots=22022.04 | 90.6 | — | |
| GLaM 62B/64EEvaluation Protocol=0-shot2022.04 | 90.3 | — | |
| GLaM 62B/64EEvaluation Protocol=1-shot2022.04 | 90.3 | — | |
| GPT-3Shots=12023.03 | 90.2 | — | |
| GPT-J# of params=6B2023.04 | 88.6 | — | |
| Alpaca# of params=7B2023.04 | 87.4 | — | |
| BloombergGPTShots=12023.03 | 82.79 | — | |
| OPT66BShots=12023.03 | 82.53 | — | |
| LaMini-LLaMA# of params=7B2023.04 | 80.8 | — | |
| BLOOM176BShots=12023.03 | 78.01 | — | |
| LaMini-GPT-J# of params=6B2023.04 | 69.2 | — | |
| GPT-NeoXShots=12023.03 | 67.86 | — | |
| MHF (H3B10K)Model Scale=100M, Embedding Parameters (Emb)=25M, Decoder Parameters (Dec)=87M, Evaluation Protocol=Zero-shot2026.06 | — | 48.21 | |
| MHF (H3B10K)Model Scale=1B, Embedding Parameters (Emb)=65M, Decoder Parameters (Dec)=0.9B, Evaluation Protocol=Zero-shot2026.06 | — | 64.51 | |
| MHF (H4B16K)Model Scale=100M, Embedding Parameters (Emb)=52M, Decoder Parameters (Dec)=87M, Evaluation Protocol=Zero-shot2026.06 | — | 50.11 | |
| MHF (H4B16K)Model Scale=1B, Embedding Parameters (Emb)=138M, Decoder Parameters (Dec)=0.9B, Evaluation Protocol=Zero-shot2026.06 | — | 64.9 | |
| MHF (H4B16K)Model Scale=3B, Embedding Parameters (Emb)=138M, Decoder Parameters (Dec)=2.8B, Evaluation Protocol=Zero-shot2026.06 | — | 64.13 | |
| Rnd. GuessModel Scale=N/A, Embedding Parameters (Emb)=-, Decoder Parameters (Dec)=-, Evaluation Protocol=Random Guessing2026.06 | — | 19.1 | |
| StandardModel Scale=100M, Embedding Parameters (Emb)=25M, Decoder Parameters (Dec)=87M, Evaluation Protocol=Zero-shot2026.06 | — | 49.47 | |
| StandardModel Scale=1B, Embedding Parameters (Emb)=65M, Decoder Parameters (Dec)=0.9B, Evaluation Protocol=Zero-shot2026.06 | — | 63.2 | |
| StandardModel Scale=3B, Embedding Parameters (Emb)=65M, Decoder Parameters (Dec)=2.8B, Evaluation Protocol=Zero-shot2026.06 | — | 62.26 | |
| Standard+2LModel Scale=1B, Embedding Parameters (Emb)=65M, Decoder Parameters (Dec)=1.0B, Evaluation Protocol=Zero-shot2026.06 | — | 63.78 | |
| Standard+4LModel Scale=100M, Embedding Parameters (Emb)=25M, Decoder Parameters (Dec)=116M, Evaluation Protocol=Zero-shot2026.06 | — | 53.08 |