NIAH
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
NIAH-Single-2 64K
100Accuracy
10
NIAH (Needle In A Haystack) Multi-task Qwen2.5-7B
96Accuracy (Single)
10
NIAH
100Score @ 4096 Context
9
NIAH Single 2
100Accuracy (1024 tokens)
8
NIAH Single-1
100Accuracy (1024)
8
NIAH 64K 60 items, 3 needle positions (test)
28.2F1 Score
8
NIAH single v1.0 (test)
100Accuracy
8
NIAH 7-client multi-needle retrieval (non-IID partition)
40MK-NIAH
7
NIAH (avg)
100Score (4k Context)
7
NIAH 256K
100Accuracy
6
NIAH
100Success Rate (8K, Single)
6
NIAH (500 samples)
34.01Latency (8K Context)
6
NIAH L=2048
100Accuracy
6
NIAH
99.8Accuracy
6
NIAH 32k
99NIAH Score
6
NIAH 16k
98.6NIAH Score
6
NIAH Needle-in-a-haystack
100NIAH Success Rate (32K Context)
6
NIAH 1
79.69Success Rate (1k Context)
5
NIAH-2 (test)
79.61NIAH-2 Success Rate (1k)
5
NIAH 32K
90.9Accuracy
4
NIAH multikey v1.0 (test)
98.7Accuracy
4
NIAH@1M RULER
99pass@1
4
NIAH Single-3
100Recall @ 32K Context
4
NIAH Single 2
1Recall @ 32K Context
4
NIAH Single-1
100Recall @ 32K
4