Spatial Reasoning on bAbI (test)
24AccuracySelf-Consistency
Evaluation results
| Method | Links | |
|---|---|---|
| Self-ConsistencyBackbone=Qwen-2.5 (14B), Reasoning Strategy=SC-5, k=52026.02 | 24 | |
| Self-ConsistencyBackbone=Gemma-3 (27B), Reasoning Strategy=SC-5, k=52026.02 | 22 | |
| Chain of Simulation (CoS)Backbone=Gemma-3 (27B), Reasoning Strategy=CoS2026.02 | 22 | |
| Chain of Simulation (CoS)Backbone=Qwen-2.5 (14B), Reasoning Strategy=CoS2026.02 | 22 | |
| DirectBackbone=Gemma-3 (27B), Reasoning Strategy=Direct2026.02 | 20 | |
| Chain-of-ThoughtBackbone=Gemma-3 (27B), Reasoning Strategy=CoT2026.02 | 20 | |
| DirectBackbone=Qwen-2.5 (14B), Reasoning Strategy=Direct2026.02 | 20 | |
| Chain-of-ThoughtBackbone=Qwen-2.5 (14B), Reasoning Strategy=CoT2026.02 | 20 | |
| Structured CoTBackbone=Qwen-2.5 (14B), Reasoning Strategy=S-CoT2026.02 | 20 | |
| Structured CoTBackbone=Gemma-3 (27B), Reasoning Strategy=S-CoT2026.02 | 18 | |
| Chain of Simulation (CoS)Backbone=LLaMA-3.1 (8B), Reasoning Strategy=CoS2026.02 | 18 | |
| Chain of Simulation (CoS)Backbone=Mistral (7B), Reasoning Strategy=CoS2026.02 | 14 | |
| DirectBackbone=LLaMA-3.1 (8B), Reasoning Strategy=Direct2026.02 | 2 | |
| Structured CoTBackbone=LLaMA-3.1 (8B), Reasoning Strategy=S-CoT2026.02 | 2 | |
| DirectBackbone=Mistral (7B), Reasoning Strategy=Direct2026.02 | 2 | |
| Chain-of-ThoughtBackbone=LLaMA-3.1 (8B), Reasoning Strategy=CoT2026.02 | 0 | |
| Self-ConsistencyBackbone=LLaMA-3.1 (8B), Reasoning Strategy=SC-5, k=52026.02 | 0 | |
| Chain-of-ThoughtBackbone=Mistral (7B), Reasoning Strategy=CoT2026.02 | 0 | |
| Structured CoTBackbone=Mistral (7B), Reasoning Strategy=S-CoT2026.02 | 0 | |
| Self-ConsistencyBackbone=Mistral (7B), Reasoning Strategy=SC-5, k=52026.02 | 0 |