Benchmarks

Task NameDataset NameSOTA ResultTrendResults

Code generation

LiveCodeBench

19.6pass@1

Code

LiveCodeBench v5

29.94Score

Code Generation

LiveCodeBench

80.1Expected Utility

Code Generation

LiveCodeBench v6

61.5Pass Rate

Code Generation

LiveCodeBench

11Pass@1

Code Generation

LiveCodeBench v5 2024.10–2025.02

60.6Pass@1

Coding

LiveCodeBench

29.1Accuracy

Coding

LiveCodeBench

52.48Score

Coding

LiveCodeBench v3

70.2Pass@1

Competitive programming

LiveCodeBench v6

0.611Solve Rate

Code

LiveCodeBench v6

67.3Pass@1

Code

LiveCodeBench

13.88Score

Code Generation

LiveCodeBench v6

94Pass@3

Code Generation

LiveCodeBench Post-March 2025 V6

94.4Pass@1 (Easy)

Code Generation

LiveCodeBench 2025/01/01 - 2025/05/01 v6 (test)

72.53Accuracy

Code Generation

LiveCodeBench

56.12Avg@3 Accuracy

Code generation

LiveCodeBench May 2023 to February 2024 (start date 2023-05-01)

25Pass@1

Code generation

LiveCodeBench May 2023 to February 2024 (start date 2023-07-01)

24.1Pass@1

Code generation

LiveCodeBench start date 2023-09-01 May 2023 to February 2024

23.2Pass@1

Code Generation

LiveCodeBench v6

43.22Avg@8

Coding

LiveCodeBench v1-5

28.3Accuracy

Code Reasoning

LiveCodeBench

92Pass@1

Code Generation

LiveCodeBench

4.93Latency (ms/tok)

Text Generation

LiveCodeBench

5.29Latency (ms/tok)

Active Search

LiveCodeBench (83 problems, 10 permutations)

179.9Average Cost