# General Reasoning on MMLU-Pro (pass@3)

Source: [General Reasoning on MMLU-Pro (pass@3)](<https://www.sota2.com/research/sota/general-reasoning-on-mmlu-pro-pass-at-3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-04T07:48:25.643Z

Evaluation of model performance on the General Reasoning task using the MMLU-Pro benchmark, measured by pass@3.

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

| Scope | Recorded value |
| --- | --- |
| Task | General Reasoning |
| Dataset (including any stated split) | MMLU-Pro |
| Metric | Pass@3 |
| Unit | PERCENT |
| Direction | Higher is better |
| Results loaded | 14 of 14 |
| Coverage | All results currently returned for this benchmark |

Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol.

## Metrics

- [Pass@3](<https://www.sota2.com/research/sota/general-reasoning-on-mmlu-pro-pass-at-3.md?metric=Pass%403>): Higher is better; unit: PERCENT.

## Recorded results

| Method | Pass@3 | Result date | Report paper |
| --- | --- | --- | --- |
| HDPO | 76.12 | Not recorded | 6a210686c10484bac23f010b |
| Critique-GRPO | 74.31 | Not recorded | 6a210686c10484bac23f010b |
| GRPO | 68.57 | Not recorded | 6a210686c10484bac23f010b |
| HDPO | 67.9 | Not recorded | 6a210686c10484bac23f010b |
| LUFFY | 67.63 | Not recorded | 6a210686c10484bac23f010b |
| HDPO | 65.63 | Not recorded | 6a210686c10484bac23f010b |
| GRPO | 59.53 | Not recorded | 6a210686c10484bac23f010b |
| Base Model | 57.95 | Not recorded | 6a210686c10484bac23f010b |
| EDGE-GRPO | 54.1 | Not recorded | 6a210686c10484bac23f010b |
| GRPO | 53.19 | Not recorded | 6a210686c10484bac23f010b |
| Base Model | 48.57 | Not recorded | 6a210686c10484bac23f010b |
| Oat | 47.72 | Not recorded | 6a210686c10484bac23f010b |
| SimpleRL | 46.48 | Not recorded | 6a210686c10484bac23f010b |
| Base Model | 36.74 | Not recorded | 6a210686c10484bac23f010b |

### HDPO

Pass@3: 76.12. Result ID: 6a212cecfbc373592c1c6e1e.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Number of samples":"16"}. This record does not assert independent reproduction.

### Critique-GRPO

Pass@3: 74.31. Result ID: 6a212cecfbc373592c1c6e1b.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7ac21e209131cea850cf.md>)

Recorded evaluation configuration: {"Backbone":"8B","Number of samples":"16"}. This record does not assert independent reproduction.

### GRPO

Pass@3: 68.57. Result ID: 6a212cecfbc373592c1c6e1d.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Number of samples":"16"}. This record does not assert independent reproduction.

### HDPO

Pass@3: 67.9. Result ID: 6a212cecfbc373592c1c6e24.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

Recorded evaluation configuration: {"Backbone":"DS-R1-Llama-8B","Number of samples":"16"}. This record does not assert independent reproduction.

### LUFFY

Pass@3: 67.63. Result ID: 6a212cecfbc373592c1c6e1a.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa79961e209131cea7e682.md>)

Recorded evaluation configuration: {"Backbone":"Qwen-Math-7B","Number of samples":"16"}. This record does not assert independent reproduction.

### HDPO

Pass@3: 65.63. Result ID: 6a212cecfbc373592c1c6e21.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

Recorded evaluation configuration: {"Backbone":"Qwen2.5-Math-7B","Number of samples":"16"}. This record does not assert independent reproduction.

### GRPO

Pass@3: 59.53. Result ID: 6a212cecfbc373592c1c6e23.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"DS-R1-Llama-8B","Number of samples":"16"}. This record does not assert independent reproduction.

### Base Model

Pass@3: 57.95. Result ID: 6a212cecfbc373592c1c6e1c.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

Recorded evaluation configuration: {"Backbone":"Qwen3-4B","Number of samples":"16"}. This record does not assert independent reproduction.

### EDGE-GRPO

Pass@3: 54.1. Result ID: 6a212cecfbc373592c1c6e17.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7a431e209131cea82028.md>)

Recorded evaluation configuration: {"Backbone":"Qwen-7B","Number of samples":"16"}. This record does not assert independent reproduction.

### GRPO

Pass@3: 53.19. Result ID: 6a212cecfbc373592c1c6e20.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa73101e209131cea5a395.md>)

[Implementation](<https://github.com/deepseek-ai/DeepSeek-Math>)

Recorded evaluation configuration: {"Backbone":"Qwen2.5-Math-7B","Number of samples":"16"}. This record does not assert independent reproduction.

### Base Model

Pass@3: 48.57. Result ID: 6a212cecfbc373592c1c6e22.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

Recorded evaluation configuration: {"Backbone":"DS-R1-Llama-8B","Number of samples":"16"}. This record does not assert independent reproduction.

### Oat

Pass@3: 47.72. Result ID: 6a212cecfbc373592c1c6e18.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7be51e209131cea8b201.md>)

Recorded evaluation configuration: {"Backbone":"Qwen2.5-Math-7B","Number of samples":"16"}. This record does not assert independent reproduction.

### SimpleRL

Pass@3: 46.48. Result ID: 6a212cecfbc373592c1c6e19.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

[Method origin paper](<https://www.sota2.com/research/paper/68fa7a721e209131cea8345b.md>)

Recorded evaluation configuration: {"Backbone":"Qwen-2.5-7B","Number of samples":"16"}. This record does not assert independent reproduction.

### Base Model

Pass@3: 36.74. Result ID: 6a212cecfbc373592c1c6e1f.

[Reporting paper](<https://www.sota2.com/research/paper/6a210686c10484bac23f010b.md>)

Recorded evaluation configuration: {"Backbone":"Qwen2.5-Math-7B","Number of samples":"16"}. This record does not assert independent reproduction.

Page 1 of 1. This is an index page, not the full collection.
