MultiArith
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
MultiArith
100Accuracy
324
MultiArith
100Accuracy
143
MultiArith (test)
99.5Accuracy
136
MultiArith
98.3Accuracy
65
MultiArith (test)
99.59Accuracy
54
MultiArith
99Original Accuracy
40
MultiArith
97.18Accuracy
33
MultiArith Out of Distribution
100Top-1 Accuracy (Maj@1)
30
MultiArith
99.3Accuracy
29
MultiArith
92Detection Accuracy
27
MultiArith
74.3Accuracy
24
MultiArith
99.15Accuracy
19
MultiArith
100Pass@1
18
MultiArith
98.5Accuracy
16
MultiArith
100Accuracy
16
MultiArith
95Accuracy
15
MultiArith (unseen)
18.33Average Success Count
12
MultiArith
43.33Accuracy
10
MultiArith
99.44Accuracy (Clean)
8
MultiArith (val)
1,640Tokens at Best Step (K)
7
MultiArith
99.64Source Library Accuracy
5
MultiArith
82.5Pass@1 Rate
5
MultiArith-100 (pilot)
100Accuracy
3
MultiArith numeric-remapping (attack set)
100Accuracy
3
MultiArith baseline original
593Correct Count
3