List your company

ResearchFormerly WizWand

NIAH

Benchmarks

TaskDatasetSOTA resultTrendResults

Single-needle retrieval

NIAH-Single-2 64K

100Accuracy

10

Long-context Retrieval

NIAH (Needle In A Haystack) Multi-task Qwen2.5-7B

96Accuracy (Single)

10

Synthetic Retrieval

NIAH

100Score @ 4096 Context

9

Long-context Retrieval

NIAH Single 2

100Accuracy (1024 tokens)

8

Long-context Retrieval

NIAH Single-1

100Accuracy (1024)

8

Needle-in-a-haystack retrieval

NIAH 64K 60 items, 3 needle positions (test)

28.2F1 Score

8

Information Retrieval

NIAH single v1.0 (test)

100Accuracy

8

Long-tail retrieval

NIAH 7-client multi-needle retrieval (non-IID partition)

40MK-NIAH

7

Long-context retrieval

NIAH (avg)

100Score (4k Context)

7

NIAH 256K

100Accuracy

6

Needle Retrieval

NIAH

100Success Rate (8K, Single)

6

Runtime efficiency

NIAH (500 samples)

34.01Latency (8K Context)

6

Needle In A Haystack retrieval

NIAH L=2048

100Accuracy

6

NIAH

99.8Accuracy

6

Long-context retrieval

NIAH 32k

99NIAH Score

6

Long-context retrieval

NIAH 16k

98.6NIAH Score

6

Needle-in-a-haystack

NIAH Needle-in-a-haystack

100NIAH Success Rate (32K Context)

6

Needle-in-a-haystack

NIAH 1

79.69Success Rate (1k Context)

5

Needle-in-a-haystack

NIAH-2 (test)

79.61NIAH-2 Success Rate (1k)

5

Long-range retrieval

NIAH 32K

90.9Accuracy

4

Information Retrieval

NIAH multikey v1.0 (test)

98.7Accuracy

4

Long Context & Context Learning

NIAH@1M RULER

99pass@1

4

Long-context recall

NIAH Single-3

100Recall @ 32K Context

4

Long-context recall

NIAH Single 2

1Recall @ 32K Context

4

Long-context recall

NIAH Single-1

100Recall @ 32K

4