Multi-step Soft Reasoning on MuSR
65.3AccuracyQwQ-32B
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| QwQ-32BModel Type=Natively-trained reasoning LM, Parameters=32B2026.06 | 65.3 | 54.1 | 73.6 | 66.5 | 80.6 | 71.3 | 77.1 | 67.2 | |
| DeepSeek-R1-8BModel Type=Distilled LRM, Parameters=8B2026.06 | 63.9 | 66.6 | 72 | 68 | 61.2 | 76.7 | 79 | 64.8 | |
| JudgeBackbone=Llama 8B2026.07 | 59.3 | — | — | — | — | — | — | — | |
| Cumulative VotingBackbone=Llama 8B2026.07 | 56.8 | — | — | — | — | — | — | — | |
| Simple VotingBackbone=Llama 8B2026.07 | 55.2 | — | — | — | — | — | — | — | |
| Ranked VotingBackbone=Llama 8B2026.07 | 52.5 | — | — | — | — | — | — | — | |
| Approval VotingBackbone=Llama 8B2026.07 | 50.9 | — | — | — | — | — | — | — | |
| Baseline with CoTBackbone=Llama 8B2026.07 | 29.5 | — | — | — | — | — | — | — | |
| Supermajority Con.Backbone=Llama 8B2026.07 | 29.3 | — | — | — | — | — | — | — | |
| Unanimity Con.Backbone=Llama 8B2026.07 | 28.2 | — | — | — | — | — | — | — | |
| Majority ConsensusBackbone=Llama 8B2026.07 | 27.8 | — | — | — | — | — | — | — | |
| BaselineBackbone=Llama 8B2026.07 | 25.8 | — | — | — | — | — | — | — |