Masked Language Modeling on C4 (val)
3.828PPLXFLASH-Quad
Evaluation results
| Method | Links | ||
|---|---|---|---|
| FLASH-QuadContext Length=40962022.02 | 3.828 | 84.9 | |
| FLASH-QuadContext Length=81922022.02 | 3.83 | 132.1 | |
| FLASH-QuadContext Length=20482022.02 | 3.864 | 61.7 | |
| Transformer++Context Length=40962022.02 | 3.876 | 143.4 | |
| FLASHContext Length=81922022.02 | 3.897 | 59.9 | |
| FLASHContext Length=40962022.02 | 3.902 | 50.7 | |
| Transformer++Context Length=20482022.02 | 3.92 | 91.6 | |
| FLASHContext Length=20482022.02 | 3.928 | 51.4 | |
| Transformer++Context Length=81922022.02 | 3.933 | 252.1 | |
| FLASH-QuadContext Length=10242022.02 | 3.964 | 50.1 | |
| Transformer+Context Length=81922022.02 | 3.986 | 252.9 | |
| Transformer+Context Length=40962022.02 | 3.989 | 142.9 | |
| FLASHContext Length=10242022.02 | 4.015 | 50.1 | |
| Transformer+Context Length=20482022.02 | 4.032 | 91.5 | |
| Transformer++Context Length=10242022.02 | 4.058 | 64.6 | |
| Transformer+Context Length=10242022.02 | 4.151 | 64.4 | |
| FLASHContext Length=5122022.02 | 4.172 | 51.2 | |
| FLASH-QuadContext Length=5122022.02 | 4.176 | 43.7 | |
| CombinerContext Length=40962022.02 | 4.195 | 68.3 | |
| TransformerContext Length=20482022.02 | 4.196 | 90.9 | |
| Transformer++Context Length=5122022.02 | 4.205 | 47.6 | |
| CombinerContext Length=81922022.02 | 4.225 | 77.3 | |
| CombinerContext Length=20482022.02 | 4.238 | 66.4 | |
| Transformer+Context Length=5122022.02 | 4.283 | 48.8 | |
| CombinerContext Length=10242022.02 | 4.317 | 66.4 | |
| TransformerContext Length=10242022.02 | 4.436 | 63.9 | |
| CombinerContext Length=5122022.02 | 4.449 | 67.2 | |
| TransformerContext Length=5122022.02 | 4.517 | 47.7 | |
| TransformerContext Length=40962022.02 | 4.602 | 142.5 | |
| TransformerContext Length=81922022.02 | 4.8766 | 252.7 | |
| PerformerContext Length=5122022.02 | 5.897 | 37.2 | |
| PerformerContext Length=10242022.02 | 6.324 | 37.6 | |
| PerformerContext Length=20482022.02 | 8.032 | 39.1 | |
| PerformerContext Length=40962022.02 | 12.622 | 36.9 | |
| PerformerContext Length=81922022.02 | 102.98 | 40.9 |