Language Modeling on Tiny Stories
2.5Perplexity (PPL)Angular (γ=8)
Evaluation Results
| Method | Links | |
|---|---|---|
| Angular (γ=8)gamma=8, Sequence Length=10242025.10 | 2.5 | |
| RACE (P=4, L=4)P=4, L=4, Sequence Length=10242025.10 | 2.6 | |
| FlashAttention2Sequence Length=10242025.10 | 2.7 | |
| RACE (P=3, L=3)P=3, L=3, Sequence Length=10242025.10 | 3.2 | |
| Linformer-128Projection Dimension=128, Sequence Length=10242025.10 | 3.7 | |
| SigmoidSequence Length=10242025.10 | 3.7 | |
| RACE (P=2, L=2)P=2, L=2, Sequence Length=10242025.10 | 4.2 | |
| LinearSequence Length=10242025.10 | 7 | |
| Performer-256Dimension=256, Sequence Length=10242025.10 | 10 |