RULER
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
RULER NIAH
89.6Formatting Compliance (1K Context)
3
RULER 1M
91.64Accuracy
3
RULER 512k
95.95Accuracy (RULER 512k)
3
RULER 256k
96.83Accuracy
3
RULER 128K
88.26Accuracy
3
RULER 250k tokens
100Single-NIAH Score
3
RULER 200k tokens
100Single-NIAH Score
3
RULER 150k tokens
100Single-NIAH
3
RULER 32K context length
15.35RULER-MV Retrieval Score
3
RULER 16K context length
29.4RULER-MV Score
3
RULER V3
100NIAH Single Query 1
3
RULER
100Performance @ 8k Context
3
RULER
100NIAH Score (8k Context)
3
RULER 256K
100NS1 (Sequence Accuracy 1)
3
RULER
83.01FullKV
2
RULER 256K
86.22Score
2
RULER transfer mixed-domain (train)
37.2Performance Score (32k Context)
2
RULER matched-training (holdout)
48.7Performance (32k Context)
2
RULER
95Score
2
RULER 32k context Average 13 tasks
0.635Score
2
RULER 32k context NIAH Single 1
1Score
2
RULER 16k context Average 13 tasks
75Score
2
RULER 16k context NIAH Single 1
1Score
2
RULER 8k context Average 13 tasks
79.3Score
2
RULER 4k context Average 13 tasks
0.842Score
2