RULER
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
RULER MRCR and NoLiMa
38.97Average Score
5
RULER
92.65RULER Accuracy (64K Context)
5
RULER 16k context length
73.2Accuracy
4
RULER 8k context length
83.8Accuracy
4
RULER 8K–64K
79.65Average Accuracy
4
RULER 4K context lengths
100MK1 Score
4
RULER 512K
89.6RULER 512K Score
4
RULER 256K
90.5Score
4
RULER 128K
94.3Score
4
RULER 64K
95Score
4
RULER 32K context slice
95.83Score
4
RULER 16K context slice
90Score
4
RULER synthetic QA (standard)
0.368F1 Score
4
RULER
6.9Delta Accuracy
4
RULER
0.015Delta Accuracy
4
RULER
83.81Accuracy (16K Context)
4
RULER synthetic 4K-64K
100Niah Score 1 (S1)
4
RULER 1.0 (test)
83.2CWE
4
RULER NIAH
100Retrieval Success (1K)
4
RULER
99.55MQ Score
4
RULER shorter context lengths ≤32K
86.43Performance Score (8K Context)
4
RULER Synthetic retrieval reasoning
71.07Accuracy (64K Context)
4
RULER (test)
92.72Retrieval (EM)
4
RULER 4K context
66.1RULER Average Score
4
RULER 128K sequences Llama3.1-70B-Instruct
65.03RULER Score
4