Character level language modeling on enwik8 (dev)
1BPCAdaptive
Evaluation results
| Method | Links | |
|---|---|---|
| Adaptive#layers=24, #Params=209M, #FLOPS=181M, Span limit S=81922019.05 | 1 | |
| Transformer-LS#Param=44M2021.07 | 1.01 | |
| Longformer#Param=41M2020.04 | 1.02 | |
| Longformer#Param=41M2021.07 | 1.02 | |
| Adaptive#layers=12, #Params=39M, #FLOPS=41M, Span limit S=81922019.05 | 1.04 | |
| Adaptive#Param=39M2020.04 | 1.04 | |
| Adaptive#Param=38M2021.07 | 1.04 | |
| Adaptive Attention SpanContext=8192, #Params=39M2023.11 | 1.04 | |
| Skip Cross-Head Transformer-XLContext=3800, #Params=41M2023.11 | 1.058 | |
| Skip Cross-Head Transformer-XLContext=1600, #Params=41M2023.11 | 1.06 |