Language Modeling on WikiText-103 byte-level (val)
1.796Loss (Context 8192)RoPE+ALiBi
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| RoPE+ALiBiEvaluation context length=81922026.05 | 1.796 | 0.028 | — | |
| Scaled-exact Jordan-RoPEEvaluation context length=8192, c=12026.05 | 1.869 | 0.031 | 0.0014 | |
| Damped RoPEEvaluation context length=81922026.05 | 1.884 | 0.051 | 0 | |
| Scaled-exact Jordan-RoPEEvaluation context length=8192, c=.1, large eta init=true2026.05 | 1.899 | 0.034 | 0.0223 | |
| Stabilized Jordan-RoPEEvaluation context length=81922026.05 | 1.903 | 0.059 | 0.0013 | |
| Scaled-exact Jordan-RoPEEvaluation context length=8192, c=.12026.05 | 1.907 | 0.047 | 0.0021 | |
| ALiBiEvaluation context length=81922026.05 | 1.932 | 0.017 | — | |
| Stabilized Jordan-RoPEEvaluation context length=8192, eta=.0052026.05 | 1.945 | 0.074 | 0.004 | |
| Stabilized Jordan-RoPEEvaluation context length=8192, eta=.0102026.05 | 1.97 | 0.097 | 0.0081 | |
| Stabilized Jordan-RoPEEvaluation context length=8192, eta=.0202026.05 | 2.131 | 0.177 | 0.0176 | |
| Stabilized Jordan-RoPEEvaluation context length=8192, eta=.0502026.05 | 2.275 | 0.219 | 0.0476 | |
| RoPEEvaluation context length=81922026.05 | 2.322 | 0.46 | — | |
| Direct-sumEvaluation context length=81922026.05 | 2.578 | 0.694 | 0.0013 |