Coding
Benchmarks
TaskDatasetSOTA resultTrendResults
Coding domain
4.14Average Wall-Clock Time (s)
40
Coding
56.1Pass@1
40
Coding Eval+ LiveCode (test)
87.2Eval+ Score
32
Coding
85F1 Score
32
Coding
63.92Original Accuracy
30
Coding Real-data 20% verified
63.92Original Accuracy
30
Coding
65TS Score
16
Coding (val)
100Pass@16
16
Coding
102.1Normalized Score
9
Coding Direct Prompt Injection
0FPR
7
Coding Qwen (n=8884)
77.74Adjusted Accuracy
5
Coding GPT-5.4
67.2Adjusted Accuracy
5
Coding Gender (test)
40Cor (%)
5
Coding Race (test)
57Correctness Rate
5
Coding Grouped General-Purpose Metrics
83.72Coding Accuracy
3
Coding Synthetic (held-out evaluation set)
0.922AUROC
3
Coding unseen tasks (test)
29.2Pass@1
3
Coding 20-task (held-out)
81.2G Score
2
Coding Hard
36.67Baseline Score
1
Coding Medium
31.96Baseline Score
1
Coding Easy
51.14Baseline
1