Benchmarks

Task NameDataset NameSOTA ResultTrendResults

Code Reasoning

LiveCodeBench v5

75.4Avg@8 Score

Coding ability

LiveCodeBench (LCB)

14.37Score

Coding Reasoning

LiveCodeBench v5

70.8Accuracy

Code Generation

LiveCodeBench v6

42.4Score

Code Generation

LiveCodeBench (LCB) Pro-Medium

28.1Pass@1

Code Generation

LiveCodeBench Pro-Easy

81.4Pass@1

Code Generation

LiveCodeBench 2408-2505

76.43Pass@1

Code Generation

LiveCodeBench v6

64.6pass@5

Code Generation

LiveCodeBench v5 (880 samples)

0.46Cost

Agentic Coding

LiveCodeBench

56Pass@1

Coding

LiveCodeBench generation_lite LeetCode easy (test)

27.4Accuracy

Reasoning

LiveCodeBench 2024-07 to 2024-12 v5

82.1Accuracy

Code Generation

LiveCodeBench (LCB)

92.88LCB Accuracy (Easy)

Coding

LiveCodeBench Pro

90.8Score

Code Generation

LiveCodeBench v5

45.34Pass@16

Win-rate attribution

LiveCodeBench 880 problems

1Tau of Mean

Code Reasoning

LiveCodeBench V5

37.6Pass@1

Code Generation

LiveCodeBench v6 (test)

37.53pass@8

Coding

LiveCodeBench Hard v6

74.6Accuracy

Coding

LiveCodeBench Medium v6

89.4Accuracy

Coding

LiveCodeBench Easy v6

99.1Accuracy

Coding

LiveCodeBench Overall v6

84.3Accuracy

Code Generation

LiveCodeBench HumanEval, HumanEval+, MBPP, MBPP+ (queries from Eurus-RL-Code) v6 (test)

43.2LCB v6 Accuracy

Code Generation

LiveCodeBench 50/50 calibration (test)

64.9CR (%)

Code Generation

LiveCodeBench

34.02Average Score