ConFiQA
Benchmarks
TaskDatasetSOTA resultTrendResults
ConFiQA MC
91.2F1 Score
42
ConFiQA (test)
95.7F1 Score
36
ConFiQA MR (test)
91.3F1 Score
36
ConFiQA 1,500 subset
81.07Ps Score
32
ConFiQA
13.21MR
24
ConFiQA
81.4In-Acc
21
ConFiQA MC 1.0 (test)
54.9Pc
20
ConFiQA MR 1.0 (test)
61.2Pc
20
ConFiQA QA 1.0 (test)
92.3Pc
20
ConFiQA
91.5Exact Match (EM)
18
ConFiQA MR 1,500
59.8Ps Score
16
ConFiQA
99Faith
12
ConFiQA (out-of-domain)
84.63Hit
12
ConFiQA-MC (Held-out)
52.8Accuracy
8
ConFiQA-QA (Held-out)
67Accuracy
8
ConFiQA MC
38.8Pc
8
ConFiQA MR
45.4Pc
8
ConFiQA-QA counterfactual contexts
81.2Accuracy
7
ConFiQA MR
89.6F1 Score
6
ConFiQA MC
53.4Ps Score
4
ConFiQA MR
54.47Ps Score
4
ConFiQA QA
74.73Ps
4