General Reasoning on Aggregate Reasoning Suite ID & OOD
98.8ID Average ScoreQwen3-8B pass@N (Upper Bound)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Qwen3-8B pass@N (Upper Bound)# Sample=-2025.11 | 98.8 | 56.1 | 72.1 | |
| Universal-PRM-Qwen2.5-Math-7B# Sample=690K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 89.7 | 44.5 | 61.5 | |
| ReProbe, Hidden States, Self-anno# Sample=32K, Input Features=Hidden States, Annotation Source=Self-anno2025.11 | 89.7 | 45.8 | 62.3 | |
| Skywork-PRM-1.5B# Sample=Unk, PRM Category=PRMs 150× Larger than ReProbes2025.11 | 89.5 | 43.8 | 60.9 | |
| Qwen2.5-Math-PRM-7B# Sample=860K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 89.2 | 44.8 | 61.5 | |
| Qwen3-14B pass@1# Sample=-2025.11 | 89 | 54.3 | 67.3 | |
| ReProbe, Attn+Logit, DeepSeek-anno# Sample=32K, Input Features=Attn+Logit, Annotation Source=DeepSeek-anno2025.11 | 89 | 47.1 | 62.8 | |
| ReProbe, Attn+Logit, Self-anno# Sample=32K, Input Features=Attn+Logit, Annotation Source=Self-anno2025.11 | 88.5 | 46.5 | 62.3 | |
| Qwen2.5-Math-7B-PRM800k# Sample=263K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 88.1 | 44.7 | 61 | |
| ReProbe, Hidden States, DeepSeek-anno# Sample=32K, Input Features=Hidden States, Annotation Source=DeepSeek-anno2025.11 | 87.6 | 46.4 | 61.9 | |
| Qwen3-8B pass@1 (Lower Bound)# Sample=-2025.11 | 87.4 | 43.3 | 59.8 | |
| RLHFlow-PRM-Mistral-Data# Sample=273K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 87.2 | 44.8 | 60.7 | |
| Math-Shepherd-PRM-7B# Sample=440K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 87.1 | 45.2 | 60.9 | |
| RLHFlow-PRM-Deepseek-Data# Sample=253K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 86.9 | 43.9 | 60 | |
| H4-Qwen2.5-PRM-1.5B-0.2# Sample=369K, PRM Category=PRMs 150× Larger than ReProbes2025.11 | 86.1 | 44.3 | 59.9 |