RULER
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
RULER 0 shot v1 (test)
94.71CWE Score
7
RULER S-NIAH
100Accuracy (S-NIAH-1, 1K Context)
6
RULER 16K context length
94.62NIAH
6
RULER 8K context length
98.5NIAH Score
6
RULER 4K context length
99.25NIAH
6
RULER 4K-32K Average
46.39CWE Score
6
RULER
10.21CWE (Context Length 4K)
6
RULER
98.62Retrieval/Reasoning Score (4K Context)
6
RULER
1.23Context Window Error (CWE)
6
RULER
90.2NIAH Score (MK, 1K)
6
RULER 8K
99.9MV Success Rate
6
RULER 4K
100MV Accuracy
6
RULER Tier 2
76.4SR
6
RULER
96.7RULER Accuracy (256k Context)
5
Ruler 4K
100Subtask 1 Score (s1)
5
Ruler (Avg.)
70.12Ruler Avg Score
5
RULER 32K
85.8Easy Average Score
5
RULER 16k context length 1.0
82.6MK1 Score
5
RULER 8k context length 1.0
76MK1 Score
5
RULER 4k context length 1.0
84.4Single-Key (mk1) Score
5
RULER
91.42RULER Score
5
RULER 11 tasks average
99.34Context Length 4K Performance
5
RULER 32k tokens
100NIAH-S
5
RULER 32k
62.28Score
5
RULER 128K context
48.38NIAH
5