Categories Companies Research

Loading the SOTA2 catalog…

Explore

Categories Companies Research

Contribute

Join SOTA2 Contact

SOTA2

© 2026 SOTA2

Build in Bellevue, USA

WizWand is now SOTA2 Research

Papers Benchmarks Datasets Tasks

CRUXEval dataset and benchmarks · SOTA2 Research

Research Datasets

CRUXEval

Benchmarks

Task NameDataset NameSOTA MethodSort

Task NameDataset NameSOTA ResultTrendResults

CRUXEval-O

83.5Accuracy

61

CRUXEval

98.8Input-CoT Accuracy

56

Code Output Prediction

CRUXEval-O

79.1Pass@1

52

Code Input Prediction

CRUXEval-I

67.5Pass@1

52

CRUXEval

76.87Accuracy

36

Execution-based output prediction

CRUXEval-O 200-sample (seed 42) (test)

55Accuracy

20

CRUXEval-O

87.5Score

19

Code Simulation

CruxEval-O

62.7Pass@1

16

Code Simulation

CruxEval-I

61.4Pass@1

16

CruxEval Output

51Score

12

Code Reasoning (Output Prediction)

CruxEval-O

36.4CX-O Score

10

Code Reasoning (Input Prediction)

CruxEval-I

35.7CX-I

10

CRUXEval I

74Accuracy

9

CRUXEval-O

41Pass@1

8

CRUXEval

88.5Pass@1

8

Output Prediction

CruxEval

94.13Pass@1

6

Reasoning failure prediction and recovery

CRUXEval L2

77Accuracy

4

CruxEval o

35.3Exact Match

4

CruxEval-i

36.2Exact Match

4

Input Prediction

CruxEval

66.5Pass@1 (inv_step_call)

3

Code Reasoning (Output Prediction)

CRUXEval-O 1-shot

84.01Accuracy

3

Code Reasoning (Input Prediction)

CRUXEval-I 1-shot

79.75Accuracy

3

Reasoning failure prediction and recovery

CRUXEval (L3)

74Accuracy

2

Reasoning failure prediction and recovery

CRUXEval L1

89Accuracy

2

Popular tasks

Code Reasoning Reasoning failure prediction and recovery Coding Code Code Reasoning (Input Prediction)

Follow for updates

Join SOTA2 Contact