MAWPS
Benchmarks
TaskDatasetSOTA resultTrendResults
MAWPS
98.5Accuracy
279
MAWPS (test)
96.5Accuracy
87
MAWPS
97.8Pass@1
38
MAWPS (5-fold cross-validation)
92.5Accuracy (5-fold)
23
MAWPS
95.94Accuracy
23
MAWPS (5-fold cross-val)
92.3Accuracy
21
MAWPS (test)
91.6Accuracy
14
MAWPS original (whole dataset)
91Value Accuracy
14
MAWPS English (test)
93Accuracy
10
MAWPS (5-fold cross val)
94.3Accuracy
10
MAWPS
73.7pass@1 Accuracy
6
MAWPS Out-of-Distribution (test)
96.5Accuracy
5
MAWPS numeric-remapping attack
98.81Accuracy
3