Code
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
Code
98.7Accuracy
242
CODE-15% (test)
97Macro AUROC
36
Code Domain-level stream
62.09AA
34
Code
0.979AUC
24
Code
138.72Throughput (tokens/s)
22
Code
65.5Code Avg
18
Code HumanEval+ LiveCodeBench v5
79.88HEval+ (Pass@1)
18
Code Category Average (test)
77.43Accuracy
18
Code
1.369End-to-end CPU Time (s)
15
CODE 15%
86.4AUC
15
Code R1
19.96Pass@1
14
Code R1
9.55Score
14
Code
52.67Performance Score
12
Code
2.88Speedup Factor
12
Code Llama-3-70B-Instruct (test)
0.951AUC
12
Code GPT-3.5 Turbo
0.906AUC
12
Code Crux, MultiPL_E, MBPP
60.12Crux Score
12
Code latest (test)
83.9HumanEval
12
Code MBPP HumanEval
76Accuracy
10
CODE n=30 (matched pairs)
85AUC
9
CODE (test)
96.79AUC
9
Code
98.3Avg Performance
9
Code Stack-Edu
2.61Speedup
8
Code Out-of-Domain
63.99Accuracy
8
CODE 15%
91.5AUC
8