RULER
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
RULER 4k context NIAH Single 1
1Score
2
RULER Sequence length = 32k
100S-NIAH Component 1 Score
2
RULER Sequence length = 16k
100S-NIAH Component 1
2
RULER niah_multikey_1 zeroshot (test)
96Accuracy
2
RULER niah_single_2 zeroshot (test)
95.5Accuracy
2
Ruler
96Retrieval Accuracy (1024 Context)
2
Ruler
88Retrieval Accuracy @ 1024 Tokens
2
Ruler
18Retrieval Accuracy (1024 Tokens)
2
Ruler
84Ruler Retrieval Acc (1024 tokens)
2
RULER 1M context length
54.19RULER Score
2
RULER 512k context length
75.12RULER Score
2
RULER 256k context length
0.799RULER Score
2
RULER 128k context length
85.13RULER Score
2
RULER 32K context length
78.5Retrieval Score
2
RULER 16K context length
82.9Retrieval Score
2
RULER 8K context length
82.3Retrieval Score
2
RULER
81.5Accuracy (8k Context)
2
RULER 4M
96S-N Score
1
RULER 2M
97S-N
1
RULER 1M
100S-N Score
1
RULER 1M
76.83Score
1
RULER 512K
84.54RULER 512K Score
1
RULER 1M
71Accuracy
1
RULER 512K
82.49Accuracy
1
Ruler
9Decoding Speedup
1