RULER
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
RULER
96.4VT Score
14
RULER
100S-NIAH-1 (1K)
13
RULER
100S1 Score
13
RULER S-NIAH-3
99.8Accuracy (512 Context)
13
RULER S-NIAH-2
100Accuracy (512 Context)
13
RULER S-NIAH-1
100Accuracy (512 tokens)
13
RULER 8K context
54.15CWE
13
RULER (test)
80Sparsity
13
Ruler NIAH Single 3
84Accuracy
13
RULER (dev)
96.1Accuracy (4K Context)
13
RULER
100Retrieval Accuracy (4K Context)
13
RULER
1.61Decoding Speedup
12
RULER 128k Zero-shot transfer
31.3Accuracy (RULER 128k Zero-shot)
12
RULER 64k Zero-shot transfer
32.9Accuracy
12
RULER 32k Zero-shot transfer
37.2Accuracy
12
RULER 128k (matched-train holdout)
36.1Accuracy
12
RULER 64k (matched-train holdout)
39.9Accuracy
12
RULER 32k (matched-train holdout)
48.7Accuracy
12
RULER 16K
85.2RULER Score
12
RULER 4K
92.97Score
12
RULER 32K
88.6Average Score (RULER 32K)
12
RULER 32768 length
91.87Accuracy
12
RULER 16384 length
92.02Accuracy
12
RULER 8192 length
93.75Accuracy
12
RULER 4096 length
94.08Accuracy
12