RULER
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
RULER
0.914RULER Score
204
RULER
100S-NIAH-1 (Pass-Key Retrieval)
94
RULER 16k (test)
93.5RULER Score
90
RULER 4k (test)
95.7RULER 4k Score
90
RULER
96.29Accuracy (8K Context)
80
RULER 32k
89.56Average Accuracy
76
RULER 16k
93.07Average Accuracy
76
RULER 8k
94.24Average Accuracy
76
RULER 4k
94.73Average Accuracy
76
RULER 16K context
83Accuracy (RULER 16K)
72
RULER Context Length = 8K
89.59Average Accuracy (RULER 8K)
72
RULER
96Score
66
RULER
157Performance @ 4K Context
65
RULER 16k
95.02Total Score
62
RULER
92.8Average Accuracy Score
59
RULER
94.7Score
49
RULER
97.36Score (4K)
49
RULER 32k
90.06Overall Score
49
RULER
94.29Score (8K Context)
48
RULER
83.01Accuracy
47
RULER
98.14Retrieval Accuracy (8K)
44
Ruler (test)
100S-NIAH-1
43
RULER 64k
100VT Score
43
RULER 32k context length
0FWE
39
RULER
97.3Accuracy
38