Character-level language modeling on text8 (dev)
1.01BPCTransformer + adaptive span
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer + adaptive span#Params=209M, Model scale=Large2019.07 | 1.01 | |
| Adaptive-SpanModel Category=Large, Span limit (S)=8192, #layers=24, Avg. span=245, #Params=209M, #FLOPS=179M2019.05 | 1.01 | |
| All-attention network + adaptive span#Params=114M, Model scale=Large2019.07 | 1.02 | |
| Transformer-LS#Param=44M2021.07 | 1.03 | |
| Longformer#Param=41M2020.04 | 1.04 | |
| Longformer#Param=41M2021.07 | 1.04 | |
| Transformer + adaptive span#Params=38M, Model scale=Small2019.07 | 1.05 | |
| All-attention network + adaptive span#Params=38M, Model scale=Small2019.07 | 1.05 | |
| Adaptive-SpanModel Category=Small, Span limit (S)=8192, #layers=12, Avg. span=314, #Params=38M, #FLOPS=42M2019.05 | 1.05 | |
| Adaptive#Param=38M2020.04 | 1.05 | |
| Adaptive#Param=38M2021.07 | 1.05 | |
| T64#Params=235M, Model scale=Large2019.07 | 1.06 | |
| T64Model Category=Large, #layers=64, Avg. span=512, #Params=235M, #FLOPS=120G2019.05 | 1.06 |