Long-context Language Modeling on RULER (16K to 512K Context Sweep)
83.81Accuracy (16K Context)MTraining
Evaluation Results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| MTrainingTraining=MTraining, Inference=Dense2025.10 | 83.81 | 75.73 | — | 64.48 | 59.47 | 54.8 | 68.07 | |
| DenseTraining=Dense, Inference=Dense2025.10 | 82.58 | 80.13 | 73.33 | 62.17 | 59.15 | 57.06 | 69.07 | |
| MTrainingTraining=MTraining, Inference=MInference2025.10 | 82.42 | 77.1 | 70.67 | 65.56 | 61.13 | 54.58 | 68.58 | |
| DenseTraining=Dense, Inference=MInference2025.10 | 55.22 | 45.41 | 25.99 | 22.4 | 19.37 | 15.19 | 30.6 |