RULER
Benchmarks
TaskDatasetSOTA resultTrendResults
RULER 32K
94.48Accuracy
38
RULER
70.7Accuracy
37
Ruler llama3-8B-Instruct (test)
100S-NIAH-1
37
RULER 64K
92.37Accuracy
37
RULER
75.3RULER Performance (8K Context)
35
RULER 8k
91.07Score
35
RULER 4k
93.73Score
35
RULER 16K
98.67Score
34
RULER-VT
99.9Accuracy
33
RULER HQA
66.86Accuracy
32
RULER 4K
95.3Accuracy
29
RULER
100Single-key Accuracy
29
RULER 128k
98Query Metric (MQ)
29
RULER 128K
88.3Accuracy
27
RULER
86.6Score
25
Ruler NIAH (Single 2)
1Accuracy
25
RULER (test)
97.6Accuracy (4K)
24
RULER QA
75Accuracy
24
RULER (test)
96.6Accuracy (4K Context)
24
RULER
92.88Performance (8K Context)
24
RULER
100VT Score
24
RULER HotpotQA
67Score
24
RULER 128K
94.67Score
23
RULER 64k context length
0FWE (Error)
22
RULER
93.4RULER Score (64K Context)
21