Language Understanding on ARC-E, BoolQ, Wino., PIQA, HellaSwag, TruthfulQA, OBQA, LogiQA
64.1ARC-E AccuracyWRAP (Synthetic+C4)
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| WRAP (Synthetic+C4)Real Tokens=85B, Model Parameters=~1.3B2024.01 | 64.1 | 62.2 | 58.9 | 75.4 | 46.2 | 40.6 | 24.1 | 23.9 | 49.4 | |
| WRAP (Synthetic)Real Tokens=85B, Model Parameters=~1.3B2024.01 | 63.9 | 60 | 58.8 | 76.1 | 45.2 | 44 | 23 | 24.1 | 49.4 | |
| C4Real Tokens=170B, Data Subset=Full, Model Parameters=~1.3B2024.01 | 61.6 | 54.2 | 59 | 74.9 | 46.8 | 33.5 | 25 | 23.4 | 47.3 | |
| RefinedWeb (RW)Real Tokens=160B, Model Parameters=~1.3B2024.01 | 61.6 | 60.7 | 57.5 | 74.3 | 45.2 | 36.8 | 21.8 | 23.2 | 47.6 | |
| C4Real Tokens=85B, Data Subset=Half, Model Parameters=~1.3B2024.01 | 61.2 | 59.1 | 57.3 | 74.9 | 46.5 | 34.1 | 22.4 | 23.5 | 47.4 | |
| RefinedWeb (RW)Real Tokens=320B, Model Parameters=~1.3B2024.01 | 60.7 | 61.1 | 57.1 | 74.4 | 45.6 | 36 | 22.6 | 22.5 | 47.5 | |
| Pythia-PileReal Tokens=300B, Model Parameters=1.4B2024.01 | 60.5 | 63.3 | 57.5 | 70.8 | 40.4 | 38.9 | 22.2 | 22.2 | 47 | |
| TinyLlamaReal Tokens=1T, Model Parameters=1.1B2024.01 | 60.3 | 57.8 | 59.1 | 73.3 | 45 | 37.6 | 21.8 | 24.5 | 47.4 |