List your company

ResearchFormerly WizWand

CodeContests

Benchmarks

TaskDatasetSOTA resultTrendResults

Code Generation

CodeContests

89.09Pass@1

68

Code Generation

CodeContests (test)

1,200Pass@1

68

Code Generation

CodeContests

21.2Accuracy

30

Code Generation

CodeContests

39.33Avg@8

26

Code Generation

CodeContests official (val)

43.6Pass@4

24

Code Generation

CodeContests

38Signal

21

Code Generation

CodeContests

55.2Pass@1

21

Code Generation

CodeContests+

39.1LCBv6 Score

15

Code Generation

CodeContests

26.7Accuracy (CC)

15

Efficiency Test Generation

CodeContests C++

62.22ASR (Fast)

8

Efficiency Test Generation

CodeContests Java

63.08Acceptance Success Rate (Fast)

8

Efficiency Test Generation

CodeContests Python

60ASR (Fast)

8

Efficiency-oriented test case generation

CodeContests

75.82ASR (Mean)

8

Code Generation

CodeContests (evaluation set)

19.7Pass@1

8

Competitive Programming

CodeContests (val)

68.86Pass@1

6

Program Synthesis

CodeContests (test)

0.205Pass@1

6

Coding Reasoning

Codecontests

65.8Pass Rate

5

Code Generation

CodeContests EASY difficulty (held-out)

34Pass@1

4

Multi-agent Selection (Pairwise Resolution)

CodeContests (test)

89.4Pairwise Resolution

3

Code Generation

CodeContests transfer

38.24Mean F1 Score

3

Competition-Level Code Generation

CodeContests (val)

2110@1k

3