AQUA
Benchmarks
TaskDatasetSOTA resultTrendResults
AQuA
93.55Accuracy
194
AQUA
85.05Accuracy
167
AQuA (test)
74.63Accuracy
72
AQUA
91.89Accuracy
65
AQuA
79.92AQuA Exact Match
60
AQuA
87.01Accuracy
59
AQUA
77.1Accuracy
57
AQuA
89.45Accuracy
43
AQuA
0.782AUROC
31
AQUA20 (test)
88.9Macro F1
28
AQUA
80.3Accuracy
26
AQuA
72CACC (%)
25
AQuA
90.16Accuracy
24
AQuA
2.02ECE
24
AQUA (test)
78.5Accuracy
24
AQUA (test)
72.44Accuracy
18
AQUA
2.96ECE
16
AQuA
74Accuracy (Without Verifier)
16
AQuA
66.36Performance (%)
12
AQuA
77Accuracy (CoT Faithfulness)
12
AQuA
28.35Accuracy
12
AQuA
96.8FRS
9
AQUA
13.76ECE
8
AQUA (val)
336Tokens at Best Step (K)
7
AQUA (test)
30.94Accuracy
6