Language Modeling on Wikipedia English (BPE 16k, context 256) (val)
16.69Validation Perplexity (PPL)D=5 concat
Evaluation results
| Method | Links | ||
|---|---|---|---|
| D=5 concatC=768, FLOPs/tok=42.5M2026.06 | 16.69 | — | |
| Roformer N=6C=768, FLOPs/tok=42.5M2026.06 | 17.95 | 1.26 | |
| D=6 addC=446, FLOPs/tok=15.9M2026.06 | 20.4 | — | |
| Roformer-hFFN N=6C=446, FLOPs/tok=15.9M2026.06 | 21.44 | 1.04 | |
| D=2 concatC=446, FLOPs/tok=7.2M2026.06 | 25.48 | — | |
| Roformer N=3C=446, FLOPs/tok=7.2M2026.06 | 27.19 | 1.71 |