Language Modeling on Toronto Book Corpus (TBC) (test)
10.72PPLStaged Training
Evaluation Results
| Method | Links | |
|---|---|---|
| Staged TrainingInference Mode=S.W., Train Speed=25.5k, Inference Speed=9.6k, Number of Parameters=261M, Subsequence length (L)=1,024, Stride (S)=5122020.12 | 10.72 | |
| Sandwich TransformerInference Mode=S.W., Train Speed=24.0k, Inference Speed=9.6k, Number of Parameters=261M, Subsequence length (L)=1,024, Stride (S)=5122020.12 | 10.83 | |
| ShortformerInference Mode=N.o., Train Speed=21.3k, Inference Speed=15.5k, Number of Parameters=261M, Subsequence length (L)=1,0242020.12 | 10.88 | |
| kNN-LMInference Mode=S.W., Train Speed=24.0k, Inference Speed=9.6k, Number of Parameters=261M, Subsequence length (L)=1,024, Stride (S)=512, Datastore Size=400GB2020.12 | 10.89 | |
| Staged TrainingInference Mode=N.o., Train Speed=25.5k, Inference Speed=19.2k, Number of Parameters=261M, Subsequence length (L)=1,0242020.12 | 11.18 | |
| PIA + CachingInference Mode=N.o., Train Speed=20.5k, Inference Speed=15.0k, Number of Parameters=261M, Subsequence length (L)=1,0242020.12 | 11.2 | |
| BaselineInference Mode=S.W., Train Speed=24.0k, Inference Speed=9.6k, Number of Parameters=261M, Subsequence length (L)=1,024, Stride (S)=5122020.12 | 11.89 | |
| BaselineInference Mode=N.o., Train Speed=24.0k, Inference Speed=19.2k, Number of Parameters=261M, Subsequence length (L)=1,0242020.12 | 12.73 |