Categories Companies Research

Loading the SOTA2 catalog…

Explore

Categories Companies Research

Contribute

Join SOTA2 Contact

SOTA2

© 2026 SOTA2

Build in Bellevue, USA

WizWand is now SOTA2 Research

Papers Benchmarks Datasets Tasks

MBPP dataset and benchmarks – Page 11 · SOTA2 Research

Research Datasets

MBPP

Benchmarks

Task NameDataset NameSOTA MethodSort

Task NameDataset NameSOTA ResultTrendResults

Code Generation

MBPP

34.2Pass@1

5

Code Optimization

MBPP-Hard In-Domain

7.1Average Calls per Task

5

MBPP base and extended (out-of-distribution)

55.83Accuracy

5

Code Generation

MBPP

314.99TPS

5

Code Generation

MBPP

25.64Pass@1

5

Code Generation

MBPP (val)

220Pass@1

5

Code Generation

MBPP+

496.19Time Per Sample (TPS)

4

Code Generation

MBPP

35.69TPS

4

Code Generation

MBPP 3-shot pass@1 (test)

30.8Pass@1 Accuracy (N=64)

4

Per-character tokenization

MBPP Sanitized

0.32Tokens/Char (MBPP Sanitized)

4

Code Generation

MBPP

61Accuracy

4

Code Generation

MBPP

55.2MBPP Accuracy (Pre-Exec)

4

MBPP (test)

523Pass Count

4

Code Generation

MBPP EvalPlus (base and plus)

46.8Base Pass Rate

4

MBPP

90.24Avg@4

4

Code Generation

MBPP+

71.5Pass@1

4

Watermark Detection

MBPP-G

100TPR

4

Code Generation

MBPP

38Accuracy

4

Code Generation

MBPP

54.6Accuracy

4

Code Generation

MBPP

70.2MBPP Score

4

Code Generation

MBPP+

100Success Rate (SR)

4

Program Synthesis

MBPP+

68Pass Rate

4

Python Code Generation

MBPP+

84Cumulative Score (CS)

4

Code Generation

MBPP

54.6MBPP Score

4

Code Test Generation

MBPP+

80.24Source Pass Rate

4

Page 11 of 13

Popular tasks

Code Generation Coding Code Watermark Detection Code Reasoning

Follow for updates

Join SOTA2 Contact