Question Answering on StrategyQA (test)
96.7Task AccuracyReProbe
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| ReProbe# Sample=32K, Input Features=Hidden States, Annotation Source=DeepSeek-anno, Decoding Method=Beam search2025.11 | 96.7 | — | — | — | — | |
| ReProbe# Sample=32K, Input Features=Attn+Logit, Annotation Source=Self-anno, Decoding Method=Beam search2025.11 | 94.8 | — | — | — | — | |
| ReProbe# Sample=32K, Input Features=Hidden States, Annotation Source=Self-anno, Decoding Method=Beam search2025.11 | 91.2 | — | — | — | — | |
| ReProbe# Sample=32K, Input Features=Attn+Logit, Annotation Source=DeepSeek-anno, Decoding Method=Beam search2025.11 | 90.7 | — | — | — | — | |
| Qwen2.5-Math-7B-PRM800k# Sample=263K, Decoding Method=Beam search2025.11 | 86.7 | — | — | — | — | |
| ToTModel=GPT-4, Prompting Strategy=Tree-of-Thought2023.05 | 83 | — | — | — | — | |
| CoTModel=GPT-4, Prompting Strategy=Chain-of-Thought2023.05 | 82 | — | — | — | — | |
| Qwen2.5-Math-PRM-7B# Sample=860K, Decoding Method=Beam search2025.11 | 79.2 | — | — | — | — | |
| LIFTBackbone=LLaMA-3-8B2025.06 | 75.85 | — | — | — | — | |
| MCR+SC@3# chains=152023.04 | 75.3 | — | — | — | — | |
| PiSSABackbone=LLaMA-3-8B2025.06 | 75.19 | — | — | — | — | |
| Full FTBackbone=LLaMA-3-8B2025.06 | 74.81 | — | — | — | — | |
| LoRABackbone=LLaMA-3-8B2025.06 | 74.44 | — | — | — | — | |
| DoRABackbone=LLaMA-3-8B2025.06 | 74.27 | — | — | — | — | |
| SC@15# chains=152023.04 | 74.1 | — | — | — | — | |
| IOModel=GPT-4, Prompting Strategy=Input-Output2023.05 | 73 | — | — | — | — | |
| DLRModel=Qwen3-8B2026.05 | 72.9 | — | — | — | — | |
| LIFTBackbone=LLaMA-2-7B2025.06 | 72.53 | — | — | — | — | |
| MCR# chains=52023.04 | 72.5 | — | — | — | — | |
| DoRABackbone=LLaMA-2-7B2025.06 | 71.98 | — | — | — | — | |
| LoRABackbone=LLaMA-2-7B2025.06 | 71.78 | — | — | — | — | |
| SC@5# chains=52023.04 | 71.4 | — | — | — | — | |
| PiSSABackbone=LLaMA-2-7B2025.06 | 71.26 | — | — | — | — | |
| Full FTBackbone=LLaMA-2-7B2025.06 | 70.61 | — | — | — | — | |
| DLR (C=1)Model=Qwen3-8B2026.05 | 70.3 | — | — | — | — | |
| DLRModel=Qwen3-4B2026.05 | 68.7 | — | — | — | — | |
| SFTModel=Qwen3-8B2026.05 | 68.2 | — | — | — | — | |
| FBackbone=T5-large, # of Params=770M, Stage=Before update2023.05 | 67.03 | -0.315 | 15.06 | 44.1 | 40.82 | |
| Iterative SocCOTModel=GPT-2 Large (774M)2022.12 | 66.4 | — | — | — | — | |
| GT FactsModel=GPT-2 XL (1.5B)2022.12 | 66.3 | — | — | — | — | |
| DLR (C=1)Model=Qwen3-4B2026.05 | 66.2 | — | — | — | — | |
| SFTModel=Qwen3-4B2026.05 | 65.8 | — | — | — | — | |
| F' (w/ Quark)Backbone=T5-large, # of Params=770M, Update Algorithm=Quark, Reward=GEN-U2023.05 | 65.06 | -0.26 | 17.01 | 40.2 | 42.79 | |
| Iterative SocCOTModel=GPT-2 XL (1.5B)2022.12 | 63.56 | — | — | — | — | |
| GT FactsModel=GPT-2 Large (774M)2022.12 | 62.8 | — | — | — | — | |
| SocGTModel=GPT-2 XL (1.5B)2022.12 | 62.3 | — | — | — | — | |
| DLRModel=Llama3.2-3B2026.05 | 62.3 | — | — | — | — | |
| Unified SocCOTModel=GPT-2 XL (1.5B)2022.12 | 62.05 | — | — | — | — | |
| SocGTModel=GPT-2 Large (774M)2022.12 | 61.32 | — | — | — | — | |
| Answer OnlyModel=GPT-2 Large (774M)2022.12 | 61.1 | — | — | — | — | |
| Answer OnlyModel=GPT-2 XL (1.5B)2022.12 | 60.51 | — | — | — | — | |
| Iterative SocCOTModel=GPT-2 Medium (355M)2022.12 | 60.31 | — | — | — | — | |
| DLRModel=Qwen3-1.7B2026.05 | 60.3 | — | — | — | — | |
| GPT-3-175BVariant=davinci-instruct-beta, # of Params=175B2023.05 | 60.04 | — | 20.3 | 25.76 | 53.93 | |
| Unified SocCOTModel=GPT-2 Large (774M)2022.12 | 59.45 | — | — | — | — | |
| CoTModel=GPT-2 XL (1.5B)2022.12 | 58.07 | — | — | — | — | |
| DLR (C=1)Model=Llama3.2-3B2026.05 | 56.1 | — | — | — | — | |
| CoTModel=GPT-2 Large (774M)2022.12 | 55.9 | — | — | — | — | |
| DLR (C=1)Model=Qwen3-1.7B2026.05 | 55.3 | — | — | — | — | |
| CoTModel=GPT-2 Medium (355M)2022.12 | 55.01 | — | — | — | — | |
| DLRModel=Qwen3-0.6B2026.05 | 54.6 | — | — | — | — | |
| Answer OnlyModel=GPT-2 Medium (355M)2022.12 | 54.1 | — | — | — | — | |
| SFTModel=Llama3.2-3B2026.05 | 52.5 | — | — | — | — | |
| SocGTModel=GPT-2 Medium (355M)2022.12 | 52.05 | — | — | — | — | |
| Unified SocCOTModel=GPT-2 Medium (355M)2022.12 | 52.05 | — | — | — | — | |
| GT FactsModel=GPT-2 Medium (355M)2022.12 | 52.02 | — | — | — | — | |
| SFTModel=Qwen3-1.7B2026.05 | 51.7 | — | — | — | — | |
| DLRModel=Llama3.2-1B2026.05 | 51.4 | — | — | — | — | |
| DLR (C=1)Model=Qwen3-0.6B2026.05 | 50.4 | — | — | — | — | |
| DLR (C=1)Model=Llama3.2-1B2026.05 | 48.6 | — | — | — | — | |
| SFTModel=Llama3.2-1B2026.05 | 48 | — | — | — | — | |
| SFTModel=Qwen3-0.6B2026.05 | 47 | — | — | — | — | |
| TokenAssortedModel=Qwen3-8B2026.05 | 41.2 | — | — | — | — | |
| TokenAssortedModel=Qwen3-4B2026.05 | 39.3 | — | — | — | — | |
| PauseTokenModel=Qwen3-8B2026.05 | 38.8 | — | — | — | — | |
| PauseTokenModel=Qwen3-4B2026.05 | 37.6 | — | — | — | — | |
| TokenAssortedModel=Llama3.2-3B2026.05 | 34.7 | — | — | — | — | |
| TokenAssortedModel=Qwen3-1.7B2026.05 | 33 | — | — | — | — | |
| PauseTokenModel=Qwen3-1.7B2026.05 | 32 | — | — | — | — | |
| TokenAssortedModel=Qwen3-0.6B2026.05 | 29.4 | — | — | — | — | |
| TokenAssortedModel=Llama3.2-1B2026.05 | 29.3 | — | — | — | — | |
| PauseTokenModel=Llama3.2-3B2026.05 | 29 | — | — | — | — | |
| PauseTokenModel=Llama3.2-1B2026.05 | 25.5 | — | — | — | — | |
| PauseTokenModel=Qwen3-0.6B2026.05 | 24.6 | — | — | — | — |