Reasoning Question Answering on StrategyQA (ACC & AUC)
80AccuracyDIRECT QUERY
Evaluation results
| Method | Links | ||
|---|---|---|---|
| DIRECT QUERYPRIVATE=NO, Backbone=GPT-4-TURBO2023.12 | 80 | 80 | |
| DIRECT QUERYPRIVATE=NO, Backbone=GPT-4O2023.12 | 79 | 78 | |
| CONFUSIONPROMPTPRIVATE=YES, Backbone=GPT-4-TURBO2023.12 | 74 | 74 | |
| CONFUSIONPROMPTPRIVATE=YES, Backbone=GPT-4O2023.12 | 73 | 74 | |
| CONFUSIONPROMPTPRIVATE=YES, Backbone=GPT-3.5-TURBO2023.12 | 72 | 73 | |
| DIRECT QUERYPRIVATE=NO, Backbone=GPT-3.5-TURBO2023.12 | 71 | 74 | |
| VICUNA-13BPRIVATE=YES2023.12 | 65 | 63 | |
| LLAMA2-7BPRIVATE=YES2023.12 | 60 | 58 | |
| PARAPHRASERPRIVATE=YES, Backbone=GPT-4-TURBO2023.12 | 56 | 53 | |
| TEXT2TEXTPRIVATE=YES, Backbone=GPT-4-TURBO2023.12 | 54 | 51 | |
| TEXT2TEXTPRIVATE=YES, Backbone=GPT-3.5-TURBO2023.12 | 53 | 51 | |
| TEXT2TEXTPRIVATE=YES, Backbone=GPT-4O2023.12 | 53 | 50 | |
| PARAPHRASERPRIVATE=YES, Backbone=GPT-4O2023.12 | 53 | 51 | |
| PARAPHRASERPRIVATE=YES, Backbone=GPT-3.5-TURBO2023.12 | 49 | 48 | |
| Phi-4 pass@N (Upper Bound)# Sample=-2025.11 | 0.991 | — | |
| RLHFlow-PRM-Deepseek-Data# Sample=253K2025.11 | 0.948 | — | |
| RLHFlow-PRM-Mistral-Data# Sample=273K2025.11 | 0.941 | — | |
| Qwen2.5-Math-7B-PRM# Sample=860K2025.11 | 0.941 | — | |
| Qwen2.5-Math-7B-PRM800k# Sample=263K2025.11 | 0.939 | — | |
| Majority Voting# Sample=-2025.11 | 0.932 | — | |
| Math-Shepherd-PRM-7B# Sample=440K2025.11 | 0.932 | — | |
| Phi-4 pass@1 (Lower Bound)# Sample=-2025.11 | 0.93 | — | |
| ReProbe, Attn+Logit, Qwen3-8B-anno# Sample=32K2025.11 | 0.93 | — | |
| Skywork-PRM-1.5B# Sample=Unk2025.11 | 0.925 | — | |
| Universal-PRM-Qwen2.5-Math-7B# Sample=690K2025.11 | 0.925 | — | |
| H4-Qwen2.5-PRM-1.5B-0.2# Sample=369K2025.11 | 0.918 | — |