Language Modeling on Wikitext-103 BB BPE Dictionary (val)
11.58PPL (Validation)Hash Layer
Evaluation results
| Method | Links | |
|---|---|---|
| Hash Layerlayers=8, modules=1x162021.06 | 11.58 | |
| Switch Transformerlayers=8, modules=1x16, load_bal=0.12021.06 | 11.67 | |
| Baseline Transformerlayers=8, d=512, D=5122021.06 | 12.58 |