PubMedQA
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
PubMedQA
1,509Token Cost (tokens/question)
11
PubMedQA PQA-L In-Domain (test)
78Accuracy
11
PubMedQA M=12 participants (test)
82.52AUC
10
PubMedQA M=8 participants (test)
84.47AUC
10
PubMedQA M=4 participants (test)
82.32AUC
10
PubMedQA (first 300 questions)
69.67Average Accuracy
10
PubMedQA (test)
75MF1 Score
10
PubMedQA
4.03Kendall's Tau
10
PubMedQA
85Accuracy
10
PubMedQA Reasoning Required
82Accuracy
10
PubMedQA
8.3PPL Delta (%)
9
PubMedQA
8.3PPL Change (%)
9
PubMedQA (test)
81.8AUROC
9
PubMedQA
97.95Context Influence Score I(D; y_tilde)
9
PubMedQA
0.807Accuracy
8
PubMedQA
68.2Accuracy
8
PubMedQA (test)
69.3Accuracy
8
PubMedQA
100AR (%)
8
PubMedQA
77.9Accuracy
8
PubMedQA
94Accuracy
7
PubMedQA Synthetic NIID 1.0 (test)
75.1Accuracy
7
PubMedQA Synthetic IID 1.0 (test)
75.1Accuracy
7
PubMedQA (PMQA) (test)
0.54AUC
7
PubMedQA
98.1Accuracy/F1
6
PubMedQA
73.6Accuracy
6