List your company

ResearchFormerly WizWand

CRUXEval

Benchmarks

TaskDatasetSOTA resultTrendResults

CRUXEval-O

83.5Accuracy

61

CRUXEval

98.8Input-CoT Accuracy

56

Code Output Prediction

CRUXEval-O

79.1Pass@1

52

Code Input Prediction

CRUXEval-I

67.5Pass@1

52

CRUXEval

76.87Accuracy

36

Execution-based output prediction

CRUXEval-O 200-sample (seed 42) (test)

55Accuracy

20

CRUXEval-O

87.5Score

19

Code Simulation

CruxEval-O

62.7Pass@1

16

Code Simulation

CruxEval-I

61.4Pass@1

16

CruxEval Output

51Score

12

Code Reasoning (Output Prediction)

CruxEval-O

36.4CX-O Score

10

Code Reasoning (Input Prediction)

CruxEval-I

35.7CX-I

10

CRUXEval I

74Accuracy

9

CRUXEval-O

41Pass@1

8

CRUXEval

88.5Pass@1

8

Output Prediction

CruxEval

94.13Pass@1

6

Reasoning failure prediction and recovery

CRUXEval L2

77Accuracy

4

CruxEval o

35.3Exact Match

4

CruxEval-i

36.2Exact Match

4

Input Prediction

CruxEval

66.5Pass@1 (inv_step_call)

3

Code Reasoning (Output Prediction)

CRUXEval-O 1-shot

84.01Accuracy

3

Code Reasoning (Input Prediction)

CRUXEval-I 1-shot

79.75Accuracy

3

Reasoning failure prediction and recovery

CRUXEval (L3)

74Accuracy

2

Reasoning failure prediction and recovery

CRUXEval L1

89Accuracy

2