Planning on Planning OOD Trips Meetings Calendar
38.7Trips Planned CountQwen3-14B pass@1
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Qwen3-14B pass@1# Sample=-2025.11 | 38.7 | 5 | 40.5 | |
| Qwen3-8B pass@N (Upper Bound)# Sample=-2025.11 | 36.2 | 16 | 60.5 | |
| ReProbe, Attn+Logit, DeepSeek-anno# Sample=32K, Input Features=Attn+Logit, Annotation Source=DeepSeek-anno2025.11 | 17.2 | 7 | 26 | |
| H4-Qwen2.5-PRM-1.5B-0.2# Sample=369K, PRM Category=PRMs 150× Larger than ReProbes2025.11 | 15.6 | 4.5 | 22 | |
| ReProbe, Hidden States, DeepSeek-anno# Sample=32K, Input Features=Hidden States, Annotation Source=DeepSeek-anno2025.11 | 15 | 5.5 | 26 | |
| ReProbe, Hidden States, Self-anno# Sample=32K, Input Features=Hidden States, Annotation Source=Self-anno2025.11 | 13.1 | 6 | 26 | |
| Universal-PRM-Qwen2.5-Math-7B# Sample=690K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 9.7 | 4 | 24 | |
| ReProbe, Attn+Logit, Self-anno# Sample=32K, Input Features=Attn+Logit, Annotation Source=Self-anno2025.11 | 9.4 | 6.5 | 31 | |
| Math-Shepherd-PRM-7B# Sample=440K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 9.1 | 4 | 30 | |
| RLHFlow-PRM-Deepseek-Data# Sample=253K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 8.7 | 3.5 | 25.5 | |
| RLHFlow-PRM-Mistral-Data# Sample=273K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 8.4 | 3.5 | 30 | |
| Qwen3-8B pass@1 (Lower Bound)# Sample=-2025.11 | 8.1 | 5.5 | 23.5 | |
| Qwen2.5-Math-PRM-7B# Sample=860K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 7.2 | 5.5 | 26.5 | |
| Skywork-PRM-1.5B# Sample=Unk, PRM Category=PRMs 150× Larger than ReProbes2025.11 | 6.9 | 6 | 23 | |
| Qwen2.5-Math-7B-PRM800k# Sample=263K, PRM Category=PRMs 750× to 810× Larger than ReProbes2025.11 | 5.9 | 6 | 27.5 |