Language Modeling on English Wikipedia
3.12PPLXFLASH-Quad
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FLASH-QuadContext Length=4096, Model Size=~110M, Hardware=32 TPU-v4 cores, Head size (s)=5122022.02 | 3.12 | 1.76 | |
| FLASHContext Length=4096, Model Size=~110M, Hardware=32 TPU-v4 cores, Head size (s)=5122022.02 | 3.23 | 2.61 | |
| FLASH-QuadContext Length=4096, Model Size=~110M, Hardware=32 TPU-v4 cores, Head size (s)=1282022.02 | 3.24 | 1.89 | |
| FLASHContext Length=4096, Model Size=~110M, Hardware=32 TPU-v4 cores, Head size (s)=512, Attention mechanism (first-to-all)=true2022.02 | 3.24 | 2.78 | |
| Transformer+Context Length=4096, Model Size=~110M, Hardware=32 TPU-v4 cores2022.02 | 3.48 | 1 | |
| CombinerContext Length=4096, Model Size=~110M, Hardware=32 TPU-v4 cores2022.02 | 3.51 | 2.78 |