SVAMP
Benchmarks
TaskDatasetSOTA resultTrendResults
SVAMP
97Accuracy
403
SVAMP (test)
94Accuracy
298
SVAMP
0MAE
198
SVAMP
96.5Accuracy
89
SVAMP
96.01Accuracy
87
SVAMP (test)
98.16Accuracy
84
SVAMP
95.2Accuracy (%)
71
SVAMP
93.7Accuracy (Overall)
54
SVAMP
80.6Mean AUROC
50
SVAMP out-of-domain (test)
97Accuracy
50
SVAMP
96.94Accuracy
38
SVAMP
94.5Value Accuracy
38
SVAMP (val)
85.1Accuracy
36
SVAMP
98.68Accuracy
35
SVAMP
93.1Pass@1
35
SVAMP
72.29Accuracy
33
SVAMP
90AUROC
32
SVAMP
5.38Average Acceptance Length
28
SVAMP
94.2Pass@1 Accuracy
27
SVAMP
92Detection Accuracy
27
SVAMP 8-shot (test)
92Accuracy
25
SVAMP
78.67Accuracy
25
SVAMP
5.7Avg Forward Passes
24
SVAMP
94.31Accuracy
21
SVAMP (full set)
71.1Exact Match Accuracy
20