Reasoning on StrategyQA (test)
100Factuality AccChain-of-Thought
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Chain-of-ThoughtBackbone=Gemma-3 (27B), Reasoning Strategy=CoT2026.02 | 100 | — | — | — | |
| Self-ConsistencyBackbone=Gemma-3 (27B), Reasoning Strategy=SC-5, k=52026.02 | 98 | — | — | — | |
| Structured CoTBackbone=Gemma-3 (27B), Reasoning Strategy=S-CoT2026.02 | 96 | — | — | — | |
| Chain of Simulation (CoS)Backbone=Gemma-3 (27B), Reasoning Strategy=CoS2026.02 | 96 | — | — | — | |
| Self-ConsistencyBackbone=LLaMA-3.1 (8B), Reasoning Strategy=SC-5, k=52026.02 | 94 | — | — | — | |
| Structured CoTBackbone=Qwen-2.5 (14B), Reasoning Strategy=S-CoT2026.02 | 94 | — | — | — | |
| Chain of Simulation (CoS)Backbone=Qwen-2.5 (14B), Reasoning Strategy=CoS2026.02 | 94 | — | — | — | |
| DirectBackbone=Gemma-3 (27B), Reasoning Strategy=Direct2026.02 | 92 | — | — | — | |
| DirectBackbone=LLaMA-3.1 (8B), Reasoning Strategy=Direct2026.02 | 92 | — | — | — | |
| Chain-of-ThoughtBackbone=Qwen-2.5 (14B), Reasoning Strategy=CoT2026.02 | 92 | — | — | — | |
| Self-ConsistencyBackbone=Qwen-2.5 (14B), Reasoning Strategy=SC-5, k=52026.02 | 92 | — | — | — | |
| DirectBackbone=Qwen-2.5 (14B), Reasoning Strategy=Direct2026.02 | 90 | — | — | — | |
| Chain of Simulation (CoS)Backbone=LLaMA-3.1 (8B), Reasoning Strategy=CoS2026.02 | 88 | — | — | — | |
| Chain-of-ThoughtBackbone=LLaMA-3.1 (8B), Reasoning Strategy=CoT2026.02 | 82 | — | — | — | |
| Structured CoTBackbone=LLaMA-3.1 (8B), Reasoning Strategy=S-CoT2026.02 | 82 | — | — | — | |
| Chain of Simulation (CoS)Backbone=Mistral (7B), Reasoning Strategy=CoS2026.02 | 82 | — | — | — | |
| DirectBackbone=Mistral (7B), Reasoning Strategy=Direct2026.02 | 76 | — | — | — | |
| Chain-of-ThoughtBackbone=Mistral (7B), Reasoning Strategy=CoT2026.02 | 74 | — | — | — | |
| Nucleus + DFDDecoding Strategy=Nucleus Sampling + Dynamic Focus Decoding2025.03 | 68.6 | 52.76 | 75.65 | 20.27 | |
| Typical + DFDDecoding Strategy=Typical Sampling + Dynamic Focus Decoding2025.03 | 68.4 | 52.81 | 76.24 | 20.29 | |
| Structured CoTBackbone=Mistral (7B), Reasoning Strategy=S-CoT2026.02 | 68 | — | — | — | |
| Top-k + DFDDecoding Strategy=Top-k Sampling + Dynamic Focus Decoding2025.03 | 67.2 | 54.52 | 78.63 | 17.54 | |
| NucleusDecoding Strategy=Nucleus Sampling2025.03 | 65.4 | 51.67 | 74.12 | 21.99 | |
| TypicalDecoding Strategy=Typical Sampling2025.03 | 65 | 51.5 | 74.1 | 22.8 | |
| Temperature + DFDDecoding Strategy=Temperature Sampling + Dynamic Focus Decoding2025.03 | 64.8 | 60.05 | 82.82 | 14.35 | |
| Top-kDecoding Strategy=Top-k Sampling2025.03 | 63.53 | 51.96 | 75.34 | 20.85 | |
| TemperatureDecoding Strategy=Temperature Sampling2025.03 | 63.47 | 56.49 | 79.44 | 16.83 | |
| Self-ConsistencyBackbone=Mistral (7B), Reasoning Strategy=SC-5, k=52026.02 | 56 | — | — | — |