Density modeling on Enwik8 (test)
0.99Bits per ByteTransformer-XL
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer-XLNumber of parameters=277M2019.04 | 0.99 | |
| Sparse TransformerNumber of parameters=95M, Attention pattern=fixed2019.04 | 0.99 | |
| Transformer-XLNumber of parameters=88M2019.04 | 1.03 | |
| Deeper Self-Attention2019.04 | 1.06 |