Reading Comprehension on RACE-h
62.3AccuracyPaLM 2-L
Evaluation results
| Method | Links | |
|---|---|---|
| PaLM 2-Lprompting=1-shot2023.05 | 62.3 | |
| PaLM 2-Mprompting=1-shot2023.05 | 57.2 | |
| PaLM 2-Sprompting=1-shot2023.05 | 53.3 | |
| PaLMprompting=1-shot2023.05 | 52.1 | |
| GPT-3 (Original)few-shot (k)=102021.08 | 41.4 | |
| EMO (Stage 5)E=64→1282026.05 | 40.39 | |
| FIXED_E=128Experts=1282026.05 | 40.31 | |
| FIXED_E=16Experts=162026.05 | 40.05 | |
| EMO (Stage 4)E=32→642026.05 | 39.71 | |
| FIXED_E=32Experts=322026.05 | 39.59 | |
| EMO (Stage 3)E=16→322026.05 | 38.51 | |
| EMO (Stage 2)E=8→162026.05 | 37.91 | |
| GPT-3 (SLW 8x Bsz)few-shot (k)=102021.08 | 37.6 | |
| EMO (Stage 1)E=82026.05 | 37.36 | |
| GPT-3 (Baseline repro)few-shot (k)=102021.08 | 35 | |
| INTRADOCChunk Length (L)=2K2024.02 | 34.49 | |
| UNIChunkChunk Length (L)=2K2024.02 | 34.01 | |
| BM25ChunkChunk Length (L)=2K2024.02 | 33.17 | |
| INTRADOCChunk Length (L)=8K2024.02 | 33.17 | |
| BM25ChunkChunk Length (L)=8K2024.02 | 32.63 | |
| MIXChunkChunk Length (L)=2K2024.02 | 32.34 | |
| UNIChunkChunk Length (L)=8K2024.02 | 31.68 | |
| MIXChunkChunk Length (L)=8K2024.02 | 31.66 | |
| +49-class Div.Class Divisions=49-class, Training Tokens=150B, Model Size=15B-A1.5B2026.02 | 28.73 | |
| +3-class Div.Class Divisions=3-class, Training Tokens=150B, Model Size=15B-A1.5B2026.02 | 28.69 | |
| Baseline MoEClass Divisions=None, Training Tokens=150B, Model Size=15B-A1.5B2026.02 | 28.34 |