Language Modeling on Toronto Book Corpus (TBC) (dev)
13.13Perplexity (PPL)Staged Training
Evaluation Results
| Method | Links | |
|---|---|---|
| Staged TrainingInference Mode=S.W., Train Speed=25.5k, Inference Speed=9.6k, Number of Parameters=261M, Subsequence length (L)=1,024, Stride (S)=5122020.12 | 13.13 | |
| ShortformerInference Mode=N.o., Train Speed=21.3k, Inference Speed=15.5k, Number of Parameters=261M, Subsequence length (L)=1,0242020.12 | 13.4 | |
| Staged TrainingInference Mode=N.o., Train Speed=25.5k, Inference Speed=19.2k, Number of Parameters=261M, Subsequence length (L)=1,0242020.12 | 13.81 | |
| PIA + CachingInference Mode=N.o., Train Speed=20.5k, Inference Speed=15.0k, Number of Parameters=261M, Subsequence length (L)=1,0242020.12 | 13.86 | |
| kNN-LMInference Mode=S.W., Train Speed=24.0k, Inference Speed=9.6k, Number of Parameters=261M, Subsequence length (L)=1,024, Stride (S)=512, Datastore Size=400GB2020.12 | 14.2 | |
| BaselineInference Mode=S.W., Train Speed=24.0k, Inference Speed=9.6k, Number of Parameters=261M, Subsequence length (L)=1,024, Stride (S)=5122020.12 | 14.75 | |
| BaselineInference Mode=N.o., Train Speed=24.0k, Inference Speed=19.2k, Number of Parameters=261M, Subsequence length (L)=1,0242020.12 | 15.38 |