Question Answering on GPQA Diamond (Pass@1)
75.7Pass@1OpenAI-o1-1217
Evaluation results
| Method | Links | |
|---|---|---|
| OpenAI-o1-12172025.01 | 75.7 | |
| DeepSeek-R1Architecture=MoE, Activated Params=37B, Total Params=671B2025.01 | 71.5 | |
| DeepSeek-R1-Distill-Llama-70BArchitecture=Llama, Parameter count=70B2025.01 | 65.2 | |
| Claude-3.5-Sonnet-10222025.01 | 65 | |
| Claude-3.5-Sonnet-10222025.01 | 65 | |
| DeepSeek-R1-Distill-Qwen-32BArchitecture=Qwen, Parameter count=32B2025.01 | 62.1 | |
| OpenAI-o1-mini2025.01 | 60 | |
| DeepSeek-V3Architecture=MoE, Activated Params=37B, Total Params=671B2025.01 | 59.1 | |
| DeepSeek-R1-Distill-Qwen-14BArchitecture=Qwen, Parameter count=14B2025.01 | 59.1 | |
| GPT-4o-05132025.01 | 49.9 | |
| GPT-4o-05132025.01 | 49.9 | |
| BF16Model=DeepSeek-R1-Distill-Qwen-7B, Bit-width=BF16, Maximum number of generation tokens=32,7682025.12 | 49.49 | |
| DeepSeek-R1-Distill-Qwen-7BArchitecture=Qwen, Parameter count=7B2025.01 | 49.1 | |
| DeepSeek-R1-Distill-Llama-8BArchitecture=Llama, Parameter count=8B2025.01 | 49 | |
| MixKVQModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=C3.4, Maximum number of generation tokens=32,7682025.12 | 45.45 | |
| KIVIModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV4, Maximum number of generation tokens=32,7682025.12 | 44.94 | |
| Low-temperatureModel=DeepSeek-Math-7B, Sampling=Low-temperature2026.01 | 43 | |
| RotateKVModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV4, Maximum number of generation tokens=32,7682025.12 | 42.93 | |
| KVQuantModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV4, Maximum number of generation tokens=32,7682025.12 | 41.41 | |
| Power SamplingModel=Qwen2.5-Math-7B2026.01 | 40.9 | |
| GRPO (MATH)Model=Qwen2.5-Math-7B2026.01 | 39.9 | |
| KVTunerModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=C3.92, Maximum number of generation tokens=32,7682025.12 | 39.9 | |
| MCMC Power SamplingModel=Qwen2.5-Math-7B2026.01 | 38.9 | |
| Power SamplingModel=DeepSeek-Math-7B2026.01 | 36.4 | |
| GRPO (MATH)Model=Qwen2.5-7B2026.01 | 35.4 | |
| KIVIModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV2, Maximum number of generation tokens=32,7682025.12 | 35.35 | |
| Low-temperatureModel=Qwen2.5-Math-7B, Sampling=Low-temperature2026.01 | 35.3 | |
| Power SamplingModel=Qwen2.5-7B2026.01 | 34.9 | |
| MCDIFFUSEModel Type=Diffusion, Parameters=7B, Slot Ordering Strategy=MCTS2026.02 | 34.78 | |
| MCMC Power SamplingModel=DeepSeek-Math-7B2026.01 | 34.5 | |
| Best-of-NModel=Qwen2.5-Math-7B, N=322026.01 | 34.3 | |
| DeepSeek-R1-Distill-Qwen-1.5BArchitecture=Qwen, Parameter count=1.5B2025.01 | 33.8 | |
| Best-of-NModel=DeepSeek-Math-7B, N=322026.01 | 33.8 | |
| ReFusion 7BModel Type=Diffusion, Parameters=7B, Slot Ordering Strategy=Sequential2026.02 | 33.43 | |
| BaseModel=DeepSeek-Math-7B2026.01 | 33.3 | |
| GRPO (MATH)Model=DeepSeek-Math-7B2026.01 | 33.3 | |
| MCMC Power SamplingModel=Qwen2.5-7B2026.01 | 31.8 | |
| Llama-3.1 8BModel Type=Autoregressive, Parameters=8B2026.02 | 31.14 | |
| ReFusion 7BModel Type=Diffusion, Parameters=7B, Slot Ordering Strategy=Vanilla2026.02 | 30.45 | |
| Low-temperatureModel=Qwen2.5-7B, Sampling=Low-temperature2026.01 | 30.3 | |
| Best-of-NModel=Qwen2.5-7B, N=322026.01 | 28.2 | |
| BaseModel=Qwen2.5-7B2026.01 | 27.8 | |
| BaseModel=Qwen2.5-Math-7B2026.01 | 27.8 | |
| ReFusion 7BModel Type=Diffusion, Parameters=7B, Slot Ordering Strategy=Random2026.02 | 27.63 | |
| KVQuantModel=DeepSeek-R1-Distill-Qwen-7B, Bit-width=KV2, Maximum number of generation tokens=32,7682025.12 | 25.76 | |
| Dream 7BModel Type=Diffusion, Parameters=7B2026.02 | 25.42 | |
| Qwen3 8BModel Type=Autoregressive, Parameters=8B2026.02 | 23.13 | |
| LLaDA 8BModel Type=Diffusion, Parameters=8B2026.02 | 21.39 | |
| Qwen2.5 7BModel Type=Autoregressive, Parameters=7B2026.02 | 16.94 |