SimpleQA
Benchmarks
TaskDatasetSOTA resultTrendResults
SimpleQA
95.3Accuracy
114
SimpleQA Verified
82.5Accuracy
77
SimpleQA
84Accuracy
50
SimpleQA
18.8Pass@1
36
SimpleQA
73.1Accuracy
34
SimpleQA Freebase-based (test)
84.8Hits@1
31
SimpleQA and 4 Hallucination Benchmarks
0.9Kendall's W
28
SimpleQA
0.039Brier Score
27
SimpleQA
95.9AUROC
27
SimpleQA
5.92Accuracy
24
SimpleQA
0.2BAS
24
SimpleQA
61.3AUROC
24
SimpleQA (train)
18.8Accuracy
21
SimpleQA
32.9Accuracy
20
SimpleQA
6.64Accuracy
20
SimpleQA
7.9ECE
18
SimpleQA-verified OOD
42.2Accuracy
18
SimpleQA-V
10Accuracy
16
SimpleQA
6.73Accuracy
16
SimpleQA
40F1 Score
14
SimpleQA
91.5Accuracy
12
SimpleQA
21.7Accuracy
11
SimpleQA
56.3Score
11
SimpleQA
53.97pass@128
11
SimpleQA out-domain (test)
36.5LasJ
11