Autoregressive Language Modeling on WikiText-103 1024 (test)
19PPLAngular
Evaluation results
| Method | Links | |
|---|---|---|
| Angulargamma=8, Sequence Length=10242025.10 | 19 | |
| RACEP=4, L=4, Sequence Length=10242025.10 | 20.9 | |
| FlashAttention2Sequence Length=10242025.10 | 20.9 | |
| RACEP=3, L=4, Sequence Length=10242025.10 | 21.5 | |
| RACEP=3, L=3, Sequence Length=10242025.10 | 21.9 | |
| RACEP=2, L=3, Sequence Length=10242025.10 | 23.4 | |
| RACEP=2, L=2, Sequence Length=10242025.10 | 23.9 |