BigCodeBench
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
BigCodeBench
73.2nDCG@1
11
BigCodeBench-I Hard
28.4Score
11
BigCodeBench-I Full
50.4Score
11
BigCodeBench Instruct Full (train)
83.3Last SR
10
BigCodeBench
4.18tau
10
BigCodeBench Full
41.58Score
10
BigCodeBench
40Score
9
BigCodeBench (hold-out)
35.8Pass@1
8
BigCodeBench Sabotage (reasoning LLM attacker)
0.87log-AUROC
8
BigCodeBench-Sabotage traditional LLM attacker
0.84log-AUROC
8
BigCodeBench
65.1Pass@1
7
BigCodeBench
24.2BigCodeBench Score
7
BigCodeBench
40.8Pass Rate
6
BigCodeBench
0.041RUT
6
BigCodeBench (test)
38.97Pass Rate
6
BigCodeBench instruction split (test)
37.86Pass Rate
6
BigCodeBench
59.5Last Epoch Success Rate
6
BigCodeBench (val)
50.8Success Rate
6
BigCodeBench Hard Set Complete
0.994Spearman's Rho
4
BigCodeBench Hard Set - Instruct
0.952Spearman Rho
4
BigCodeBench Complete (Full Set)
0.943Spearman Rho
4
BigCodeBench Instructions-Only
1Spearman rho
4
BigCodeBench
54Pass Rate
3
BigCodeBench
63.9Pass@1
3
BigCodeBench
41.67Pass@1
3