Language Modeling on C4 (Comprehensive Evaluation Suite)
10.9Perplexity (PPL)MoEUT
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| MoEUTnumber of parameters=1040M, zero-shot=true2024.05 | 10.9 | 38.4 | 81.6 | 89.2 | 43.7 | 69.9 | 41.3 | 60.7 | |
| Baselinenumber of parameters=1040M, zero-shot=true2024.05 | 11.15 | 38.4 | 81.2 | 89 | 42 | 68.6 | 39.7 | 59.8 | |
| MoEUTnumber of parameters=728M, zero-shot=true2024.05 | 11.34 | 36 | 80.8 | 88.4 | 41.8 | 69.2 | 39.6 | 59.3 | |
| Baselinenumber of parameters=728M, zero-shot=true2024.05 | 11.59 | 37.8 | 80.7 | 88.2 | 40.5 | 67.7 | 39.3 | 59 | |
| MoEUTnumber of parameters=319M, zero-shot=true2024.05 | 12.65 | 30.8 | 80.2 | 86.9 | 37.3 | 67 | 37.3 | 56.6 | |
| Baselinenumber of parameters=319M, zero-shot=true2024.05 | 12.81 | 33.3 | 78.5 | 87.2 | 36.1 | 67.1 | 37.2 | 56.6 | |
| MoEUTnumber of parameters=244M, zero-shot=true2024.05 | 13.24 | 30.6 | 79.7 | 85.3 | 35.7 | 65.2 | 36.4 | 55.5 | |
| Baselinenumber of parameters=244M, zero-shot=true2024.05 | 13.4 | 33.1 | 78.5 | 86 | 34.5 | 64.9 | 36.9 | 55.6 | |
| MoEUTnumber of parameters=126M, zero-shot=true2024.05 | 14.76 | 27.2 | 79.4 | 84.2 | 32.3 | 64.4 | 35.3 | 53.8 | |
| Baselinenumber of parameters=126M, zero-shot=true2024.05 | 14.97 | 28.5 | 77 | 84.4 | 31.7 | 62.7 | 35.2 | 53.2 | |
| MoEUTnumber of parameters=44M, zero-shot=true2024.05 | 18.3 | 23.2 | 78.2 | 81.1 | 29.2 | 61.3 | 33.5 | 51.1 | |
| Baselinenumber of parameters=44M, zero-shot=true2024.05 | 18.97 | 21.9 | 73.5 | 81.3 | 28.3 | 59.9 | 31.7 | 49.4 |