Language Modeling on Synthetic query
96.9Accuracy (1024 tokens)RoPE
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| RoPETraining steps=12002026.05 | 96.9 | 83.3 | 78.1 | |
| Stabilized Jordan-RoPETraining steps=12002026.05 | 96.9 | 93.8 | 90.6 | |
| Scaled-exactTraining steps=1200, c=0.12026.05 | 95.8 | 91.7 | 86.5 | |
| Scaled-exactTraining steps=1200, c=12026.05 | 95.8 | 75 | 52.1 | |
| Damped RoPETraining steps=12002026.05 | 93.8 | 91.7 | 86.5 | |
| RoPE+ALiBiTraining steps=12002026.05 | 92.7 | 52.1 | 53.1 | |
| Scaled-exactTraining steps=1200, c=0.1, initialization=large eta init2026.05 | 91.7 | 92.7 | 85.4 | |
| Direct-sumTraining steps=12002026.05 | 59.4 | 59.4 | 50 |