RULER
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
RULER 128K
74.8Accuracy
4
RULER 64K
88.4Accuracy
4
RULER 32K
94.6Accuracy
4
RULER 16K
97.1Accuracy
4
RULER 8K
98.2Accuracy
4
RULER 4K
98.8Accuracy
4
RULER 16k context 3k budget
100S-1 Score
4
RULER S-NIAH-2 OOD
83.2Success Rate (4K Context)
4
RULER S-NIAH-2 (ID)
100Retrieval Success Rate (1K)
4
RULER S-NIAH-1 OOD
100Success Rate (4K Context)
4
RULER S-NIAH-1 ID
100Retrieval Success Rate (1K Context)
4
RULER (test)
95.94RULER Accuracy (4K)
4
RULER 8K context length
66.5Avg Task Score
4
RULER 64K context length
84.29Accuracy
4
RULER ultra-long context official
96Accuracy (128K)
4
RULER 512K
99S-N Score
3
RULER 128K
99S-N Score
3
RULER 32K context
46.98RULER Score
3
RULER 32K context length
100S1
3
RULER 1200 (test)
84.1Success Rate
3
RULER 8k
60SQuAD Score
3
RULER 32K context length
92.78NIAH
3
RULER 16K context length
95.6NIAH-MV
3
RULER 8K context length
95.4NIAH-MV
3
RULER 4K context length
98.9NIAH-MV
3