Language Modeling on pushshift.io Reddit (test)
22.78PerplexityDeeper Transformer (more compute)
Evaluation results
| Method | Links | |
|---|---|---|
| Deeper Transformer (more compute)Configuration=layers=22, d=1536, D=4096, Params=755M2021.06 | 22.78 | |
| Switch TransformerConfiguration=layers=11,modules=5x16, load_bal=0.1, Params=852M2021.06 | 22.93 | |
| Hash LayerConfiguration=layers=11,modules=1x128, Params=1.28B2021.06 | 22.95 | |
| Hash LayerConfiguration=layers=11,modules=1x64, Params=751M2021.06 | 23.23 | |
| Switch TransformerConfiguration=layers=11,modules=5x16, load_bal=0.01, Params=852M2021.06 | 23.25 | |
| Hash LayerConfiguration=layers=11,modules=5x16, Params=852M2021.06 | 23.27 | |
| Wider Transformer (more compute)Configuration=layers=11, d=2048, D=6144, Params=755M2021.06 | 23.38 | |
| Switch TransformerConfiguration=layers=11,modules=1x128, load_bal=0.1, Params=1.28B2021.06 | 23.58 | |
| Switch TransformerConfiguration=layers=11,modules=1x64, load_bal=0.1, Params=751M2021.06 | 23.73 | |
| Baseline TransformerConfiguration=layers=11, d=1024, D=4096, Params=222M2021.06 | 24.96 |