Language Modeling on MADLAD-400 (val)
21.1PPL (ar)Branch-Train-Mix
Evaluation results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Branch-Train-MixParameters=1.7B2025.06 | 21.1 | 15.3 | 29.7 | 13.1 | 9.4 | 14.1 | 18.6 | 3.1 | 5.3 | 10.9 | 4.5 | 5.4 | 2.6 | 22.4 | 25.7 | 26.2 | 24.2 | 29.8 | 7.5 | 24.1 | 14.3 | |
| DMOEParameters=1.7B2025.06 | 22.7 | 14.9 | 31.3 | 12.8 | 8.9 | 14.7 | 19.1 | 3 | 5.1 | 11.3 | 4.3 | 5.1 | 2.5 | 22.1 | 26 | 23.9 | 23.2 | 29.2 | 6.8 | 22.8 | 13.7 | |
| BLOOM1.7B + Pre-trainParameters=1.7B, Training=Continued Pre-training2025.06 | 22.9 | 15.9 | 30.3 | 14.1 | 9.7 | 14.7 | 19.3 | 3.1 | 5.4 | 11.3 | 4.5 | 5.4 | 2.5 | 22.3 | 26.8 | 27.9 | 23.9 | 30.8 | 7.4 | 26 | 14.9 | |
| BLOOM1.7BParameters=1.7B2025.06 | 30.4 | 56.4 | 45.3 | 44.9 | 35 | 20.8 | 27.8 | 9.7 | 26.7 | 15.3 | 19.1 | 21.1 | 4.4 | 46.8 | 44 | 113.2 | 61.7 | 80.3 | 28.4 | 260.8 | 71.9 | |
| DMOEParameters=560M2025.06 | 32.1 | 19.5 | 40.8 | 16.3 | 11.3 | 19.8 | 25.8 | 3.4 | 6.4 | 14.7 | 5.2 | 6.2 | 2.7 | 29.3 | 35.2 | 31.4 | 30.5 | 39.7 | 8.1 | 28.3 | 17.7 | |
| Branch-Train-MixParameters=560M2025.06 | 32.5 | 21 | 40.3 | 17.1 | 12.1 | 20 | 26.1 | 3.5 | 6.6 | 14.6 | 5.4 | 6.3 | 2.8 | 31.6 | 37.1 | 35.4 | 34.1 | 43.5 | 9 | 31.2 | 19.2 | |
| BLOOM560M + Pre-trainParameters=560M, Training=Continued Pre-training2025.06 | 34.4 | 23.8 | 44.6 | 20.5 | 12.9 | 20.5 | 26.9 | 3.6 | 7.3 | 15.1 | 5.7 | 6.8 | 2.8 | 30.3 | 37.6 | 40.2 | 32.5 | 45.5 | 9.2 | 36.2 | 20.7 | |
| BLOOM560MParameters=560M2025.06 | 42.4 | 111.3 | 66.8 | 82.4 | 55.5 | 30.6 | 41.3 | 13.7 | 44.5 | 21.8 | 29 | 31.2 | 6.1 | 91.4 | 72.5 | 261.7 | 131.6 | 224.9 | 44.5 | 1,278.9 | 154.4 |