SVAMP
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
SVAMP
83.8Accuracy (SVAMP)
20
SVAMP
0.621AUROC
20
SVAMP English (test)
67.8Accuracy
20
SVAMP
92.33Accuracy
19
SVAMP
84.3Accuracy
16
SVAMP
97Pass@5
16
SVAMP
56.8Accuracy
15
SVAMP
97.61AUC
15
SVAMP
94.1Accuracy
15
SVAMP
94.3Pass@1 Accuracy
14
SVAMP
94.2Accuracy (%)
14
SVAMP
83.33Accuracy
14
SVAMP (held-out)
78.3Performance
14
SVAMP latest (test)
64.8Accuracy
14
SVAMP
96Accuracy (%)
12
SVAMP
7,906Max Output Length
12
SVAMP
79.53Accuracy
12
SVAMP
97.33Verifiability Score
12
SVAMP
71.11Reusability Score
12
SVAMP
69.3Accuracy
12
SVAMP
94Accuracy
11
SVAMP
96.7Accuracy
10
SVAMP
4.96Average Acceptance Length
10
SVAMP
90.33Accuracy (SVAMP)
9
SVAMP
0.893Accuracy (Context Size 128)
9