PopQA
Benchmarks
TaskDatasetSOTA resultTrendResults
PopQA OOD
45.7Exact Match Accuracy
15
PopQA Gemini outputs (test)
78.7AUROC
15
PopQA GPT outputs (test)
0.802AUROC
15
PopQA Llama outputs (test)
0.77AUROC
15
PopQA out-of-domain (val test)
50.1Exact Match (EM)
15
PopQA
45.5Accuracy (EM)
14
POPQA
65.04fEM
14
PopQA 2018 Wikipedia dump (dev)
47.2Accuracy
14
POPQA Long-tail
74.5Recall@10
14
PopQA
62.545Accuracy
13
PopQA Pop.
27.7OSR Score
13
PopQA
46.3Exact Match (EM)
13
PopQA (test)
0.859AUROC
12
PopQA held-out 50% (test)
85.7AUROC
12
PopQA
47.4Accuracy
12
PopQA
66.3CEM (%)
12
PopQA
50.6Accuracy
11
PopQA (test)
53.8Recall@5
11
PopQA (test)
51.26EM Accuracy
11
PopQA
65.7Accuracy
11
PopQA (test)
0.435r_connect
10
PopQA D_special (test)
48.25r_pub Score
10
PopQA
47.87EM
10
PopQA
28.7EM
10
PopQA 1,000 queries (test)
50.94EM
10