Language Modeling on pushshift.io Reddit (val)
22.72PPLDeeper Transformer (more compute)
Evaluation results
| Method | Links | |
|---|---|---|
| Deeper Transformer (more compute)Configuration=layers=22, d=1536, D=4096, Params=755M2021.06 | 22.72 | |
| Hash LayerConfiguration=layers=11,modules=1x128, Params=1.28B2021.06 | 22.89 | |
| Switch TransformerConfiguration=layers=11,modules=5x16, load_bal=0.1, Params=852M2021.06 | 23 | |
| Hash LayerConfiguration=layers=11,modules=1x64, Params=751M2021.06 | 23.16 | |
| Switch TransformerConfiguration=layers=11,modules=5x16, load_bal=0.01, Params=852M2021.06 | 23.19 | |
| Hash LayerConfiguration=layers=11,modules=5x16, Params=852M2021.06 | 23.21 | |
| Wider Transformer (more compute)Configuration=layers=11, d=2048, D=6144, Params=755M2021.06 | 23.32 | |
| Switch TransformerConfiguration=layers=11,modules=1x128, load_bal=0.1, Params=1.28B2021.06 | 23.52 | |
| Switch TransformerConfiguration=layers=11,modules=1x64, load_bal=0.1, Params=751M2021.06 | 23.65 | |
| Baseline TransformerConfiguration=layers=11, d=1024, D=4096, Params=222M2021.06 | 24.9 |