Long-Context Language Modeling on RULER (4k and 8k Component Evaluation)
10.21CWE (Context Length 4K)RoPE
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| RoPEModel=nanoGPT-44M2026.05 | 10.21 | 11.52 | 12.19 | 12.59 | 8.45 | 8.95 | 10.43 | 11.11 | |
| p-RoPEModel=nanoGPT-44M2026.05 | 9.97 | 10.8 | 11.98 | 12.06 | 8.63 | 8.99 | 9.87 | 10.68 | |
| GAPEModel=nanoGPT-44M2026.05 | 9.32 | 9.48 | 10.59 | 11.36 | 8.14 | 8.8 | 9.18 | 10.4 | |
| RoPEModel=GPT2-124M2026.05 | 8.21 | 9.13 | 9.51 | 10.57 | 6.45 | 7.54 | 9.9 | 10.7 | |
| p-RoPEModel=GPT2-124M2026.05 | 8.1 | 8.82 | 9.26 | 10.05 | 6.32 | 6.88 | 8.5 | 9.53 | |
| GAPEModel=GPT2-124M2026.05 | 7.96 | 8.83 | 8.94 | 9.87 | 6.39 | 6.87 | 8.31 | 8.98 |