Logical Reasoning on ProofWriter (Accuracy)
75AccuracySoftCoT
Evaluation Results
| Method | Links | |
|---|---|---|
| SoftCoTBase model=Qwen2.5, Transfer cost=full retrain2026.05 | 75 | |
| HyperGuideBase model=Qwen2.5, Transfer cost=small MLP2026.05 | 75 | |
| Self-ConsistencyBase model=Qwen2.5, Transfer cost=N/A2026.05 | 74 | |
| OVMBase model=Mistral, Transfer cost=full retrain2026.05 | 72 | |
| Few-shotBase model=Qwen2.5, Transfer cost=N/A2026.05 | 70.4 | |
| Tree of ThoughtsBase model=Qwen2.5, Transfer cost=N/A2026.05 | 69 | |
| HyperGuideBase model=Mistral, Transfer cost=small MLP2026.05 | 69 | |
| ORACLEBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 68.2 | |
| Self-ConsistencyBase model=Mistral, Transfer cost=N/A2026.05 | 67.6 | |
| ORACLEBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 67.5 | |
| Tree of ThoughtsBase model=Mistral, Transfer cost=N/A2026.05 | 66 | |
| ORACLEBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 64.5 | |
| self-rewardingBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 64.5 | |
| CoTBackbone=Qwen-2.5-7B-instruct, Setting=four-shot2026.03 | 63.8 | |
| RFTBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 63.5 | |
| self-rewardingBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 63.4 | |
| RFTBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 62 | |
| SoftCoTBase model=GPT-OSS, Transfer cost=full retrain2026.05 | 61.4 | |
| CoTBackbone=Llama-3.1-8B-Instruct, Setting=four-shot2026.03 | 60.3 | |
| CoTBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 60 | |
| HyperGuideBase model=GPT-OSS, Transfer cost=small MLP2026.05 | 59 | |
| self-rewardingBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 58.8 | |
| ToT-SFTBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 57 | |
| Few-shotBase model=Mistral, Transfer cost=N/A2026.05 | 57 | |
| SoftCoTBase model=Mistral, Transfer cost=full retrain2026.05 | 57 | |
| ToT-SFTBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 54.8 | |
| ToT-SFTBackbone=Qwen-2.5-7B-instruct, Setting=zero-shot2026.03 | 54.8 | |
| CoTBackbone=Llama-3.1-8B-Instruct, Setting=zero-shot2026.03 | 53.7 | |
| RFTBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 53.6 | |
| ToTReasoning Method=Tree-of-Thought, Evaluation Protocol=String Match (SM)2026.03 | 51.7 | |
| CoTBackbone=Mistral-7B-Instruct-v0.3, Setting=four-shot2026.03 | 50.5 | |
| NoTReasoning Method=Network-of-Thought, Evaluation Protocol=LLM-as-Judge2026.03 | 50.3 | |
| NoTReasoning Method=Network-of-Thought, Evaluation Protocol=String Match (SM)2026.03 | 49 | |
| PT-SFTBase model=Qwen2.5, Transfer cost=full retrain2026.05 | 49 | |
| Self-ConsistencyBase model=GPT-OSS, Transfer cost=N/A2026.05 | 47 | |
| Tree of ThoughtsBase model=GPT-OSS, Transfer cost=N/A2026.05 | 46 | |
| CoTReasoning Method=Chain-of-Thought, Evaluation Protocol=String Match (SM)2026.03 | 43.3 | |
| PT-SFTBase model=Mistral, Transfer cost=full retrain2026.05 | 42.6 | |
| PT-SFTBase model=GPT-OSS, Transfer cost=full retrain2026.05 | 42.4 | |
| OVMBase model=Qwen2.5, Transfer cost=full retrain2026.05 | 38 | |
| CoTBackbone=Mistral-7B-Instruct-v0.3, Setting=zero-shot2026.03 | 36.3 | |
| OVMBase model=GPT-OSS, Transfer cost=full retrain2026.05 | 33 | |
| Few-shotBase model=GPT-OSS, Transfer cost=N/A2026.05 | 29.6 |