Language Modeling on Language Pre-training Dataset (test)
2.341Final LossSMART
Evaluation results
| Method | Links | |
|---|---|---|
| SMARTParameters (M)=45.512025.12 | 2.341 | |
| GPT-2Parameters (M)=124.42025.12 | 2.787 | |
| Pure TransformerParameters (M)=52.02025.12 | 3.456 | |
| DistilBERTParameters (M)=89.82025.12 | 10.43 | |
| BERTParameters (M)=133.02025.12 | 10.46 |