Question Answering on Specialized Knowledge Tasks (ARC-C, SciQ, PubMedQA, MathQA, MMLU) zero-shot
29.9ARC-CWRAP (Synthetic+C4)
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| WRAP (Synthetic+C4)Real Tokens=85B, Model Size=~1.3B, Runs=32024.01 | 29.9 | 87.6 | 61.5 | 23.9 | 24.8 | 45.5 | |
| WRAP (Synthetic)Real Tokens=85B, Model Size=~1.3B, Runs=32024.01 | 29.7 | 87 | 60.2 | 23.4 | 24.6 | 45 | |
| RW (RefinedWeb)Real Tokens=320B, Model Size=~1.3B2024.01 | 27.8 | 88 | 57.4 | 23 | 25.4 | 44.3 | |
| TinyLlamaReal Tokens=1T, Model Size=~1.3B2024.01 | 27.8 | 88.9 | 61.4 | 24.1 | 25.8 | 45.6 | |
| RW (RefinedWeb)Real Tokens=160B, Model Size=~1.3B2024.01 | 27.2 | 87.2 | 56.2 | 24.1 | 25.9 | 44.1 | |
| Full C4Real Tokens=170B, Model Size=~1.3B2024.01 | 26.8 | 85 | 57.4 | 24.3 | 23.9 | 43.5 | |
| Half C4Real Tokens=85B, Model Size=~1.3B2024.01 | 26.3 | 84.5 | 57.2 | 23.4 | 24.2 | 43.1 | |
| Pythia-PileReal Tokens=300B, Model Size=~1.3B2024.01 | 26.1 | 86.6 | 60.6 | 25.2 | 24.3 | 44.6 |