RULER
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
RULER 128K context
66.71Accuracy
12
RULER 32K context length (test)
100Niah1 Score
12
RULER 8k
79.28F1 Score
12
RULER 4k
81.8F1 Score
12
RULER SQuAD
81.25F1 Score (32K Context)
11
RULER-HQA 56K
81.25Normalized Sub-EM Accuracy
11
RULER HQA 28K context length
82.03Normalized Sub-EM Accuracy
11
RULER 16K
100S-N Score
10
RULER 8K
100S-N
10
RULER HotpotQA
82.38Accuracy @ 7K Context
10
RULER 32M
85.45S-N Score
10
RULER 8M
79.21S-N
10
RULER 1M
89.09S-N Score
10
RULER 16k context
100S1 Accuracy
10
RULER 128k
89.49Accuracy
10
RULER
96.74Retrieval Accuracy (4K)
10
RULER (test)
96.74Accuracy (4K Context)
10
RULER 16k
70.56F1 Score
10
RULER-32K (test)
100SG-1 Score
9
RULER 64K context 1.0 (test)
100N-S Score 1
9
RULER 8k context NIAH Single 1
100Accuracy
9
RULER 8k branch-stress check
0.02NIAH Single-Key Score
9
RULER
100S1 Score
9
RULER S-NIAH
100S-NIAH PK Success Rate (2K)
9
RULER-PT (aggregate)
97.7RULER-PT (Aggregate) Score @ 1024 Context
9