# Reasoning Question Answering on StrategyQA (ACC & AUC) Source: [Reasoning Question Answering on StrategyQA (ACC & AUC)]() Collection index: [llms.txt]() Data updated: 2026-04-23T10:01:14.484Z Evaluation of model performance on the StrategyQA dataset for Reasoning Question Answering, reporting Accuracy (ACC) and Area Under the Curve (AUC). A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Reasoning Question Answering | | Dataset (including any stated split) | StrategyQA | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 26 of 26 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. - [AUC](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | DIRECT QUERY | 80 | Not recorded | 68fa758c1e209131cea68880 | | DIRECT QUERY | 79 | Not recorded | 68fa758c1e209131cea68880 | | CONFUSIONPROMPT | 74 | Not recorded | 68fa758c1e209131cea68880 | | CONFUSIONPROMPT | 73 | Not recorded | 68fa758c1e209131cea68880 | | CONFUSIONPROMPT | 72 | Not recorded | 68fa758c1e209131cea68880 | | DIRECT QUERY | 71 | Not recorded | 68fa758c1e209131cea68880 | | VICUNA-13B | 65 | Not recorded | 68fa758c1e209131cea68880 | | LLAMA2-7B | 60 | Not recorded | 68fa758c1e209131cea68880 | | PARAPHRASER | 56 | Not recorded | 68fa758c1e209131cea68880 | | TEXT2TEXT | 54 | Not recorded | 68fa758c1e209131cea68880 | | TEXT2TEXT | 53 | Not recorded | 68fa758c1e209131cea68880 | | TEXT2TEXT | 53 | Not recorded | 68fa758c1e209131cea68880 | | PARAPHRASER | 53 | Not recorded | 68fa758c1e209131cea68880 | | PARAPHRASER | 49 | Not recorded | 68fa758c1e209131cea68880 | | Phi-4 pass@N (Upper Bound) | 0.991 | Not recorded | 6912c2ee2828b7cc6937d9ec | | RLHFlow-PRM-Deepseek-Data | 0.948 | Not recorded | 6912c2ee2828b7cc6937d9ec | | RLHFlow-PRM-Mistral-Data | 0.941 | Not recorded | 6912c2ee2828b7cc6937d9ec | | Qwen2.5-Math-7B-PRM | 0.941 | Not recorded | 6912c2ee2828b7cc6937d9ec | | Qwen2.5-Math-7B-PRM800k | 0.939 | Not recorded | 6912c2ee2828b7cc6937d9ec | | Majority Voting | 0.932 | Not recorded | 6912c2ee2828b7cc6937d9ec | | Math-Shepherd-PRM-7B | 0.932 | Not recorded | 6912c2ee2828b7cc6937d9ec | | Phi-4 pass@1 (Lower Bound) | 0.93 | Not recorded | 6912c2ee2828b7cc6937d9ec | | ReProbe, Attn+Logit, Qwen3-8B-anno | 0.93 | Not recorded | 6912c2ee2828b7cc6937d9ec | | Skywork-PRM-1.5B | 0.925 | Not recorded | 6912c2ee2828b7cc6937d9ec | | Universal-PRM-Qwen2.5-Math-7B | 0.925 | Not recorded | 6912c2ee2828b7cc6937d9ec | | H4-Qwen2.5-PRM-1.5B-0.2 | 0.918 | Not recorded | 6912c2ee2828b7cc6937d9ec | ### DIRECT QUERY Accuracy: 80. Result ID: 69d837cb47f34ce924c000a8. [Reporting paper]() Recorded evaluation configuration: {"PRIVATE":"NO","Backbone":"GPT-4-TURBO"}. This record does not assert independent reproduction. ### DIRECT QUERY Accuracy: 79. Result ID: 69d837cb47f34ce924c000a9. [Reporting paper]() Recorded evaluation configuration: {"PRIVATE":"NO","Backbone":"GPT-4O"}. This record does not assert independent reproduction. ### CONFUSIONPROMPT Accuracy: 74. Result ID: 69d837cb47f34ce924c000ab. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"PRIVATE":"YES","Backbone":"GPT-4-TURBO"}. This record does not assert independent reproduction. ### CONFUSIONPROMPT Accuracy: 73. Result ID: 69d837cb47f34ce924c000ac. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"PRIVATE":"YES","Backbone":"GPT-4O"}. This record does not assert independent reproduction. ### CONFUSIONPROMPT Accuracy: 72. Result ID: 69d837cb47f34ce924c000aa. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"PRIVATE":"YES","Backbone":"GPT-3.5-TURBO"}. This record does not assert independent reproduction. ### DIRECT QUERY Accuracy: 71. Result ID: 69d837cb47f34ce924c000a7. [Reporting paper]() Recorded evaluation configuration: {"PRIVATE":"NO","Backbone":"GPT-3.5-TURBO"}. This record does not assert independent reproduction. ### VICUNA-13B Accuracy: 65. Result ID: 69d837cb47f34ce924c000a0. [Reporting paper]() Recorded evaluation configuration: {"PRIVATE":"YES"}. This record does not assert independent reproduction. ### LLAMA2-7B Accuracy: 60. Result ID: 69d837cb47f34ce924c0009f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"PRIVATE":"YES"}. This record does not assert independent reproduction. ### PARAPHRASER Accuracy: 56. Result ID: 69d837cb47f34ce924c000a5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"PRIVATE":"YES","Backbone":"GPT-4-TURBO"}. This record does not assert independent reproduction. ### TEXT2TEXT Accuracy: 54. Result ID: 69d837cb47f34ce924c000a2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"PRIVATE":"YES","Backbone":"GPT-4-TURBO"}. This record does not assert independent reproduction. ### TEXT2TEXT Accuracy: 53. Result ID: 69d837cb47f34ce924c000a1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"PRIVATE":"YES","Backbone":"GPT-3.5-TURBO"}. This record does not assert independent reproduction. ### TEXT2TEXT Accuracy: 53. Result ID: 69d837cb47f34ce924c000a3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"PRIVATE":"YES","Backbone":"GPT-4O"}. This record does not assert independent reproduction. ### PARAPHRASER Accuracy: 53. Result ID: 69d837cb47f34ce924c000a6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"PRIVATE":"YES","Backbone":"GPT-4O"}. This record does not assert independent reproduction. ### PARAPHRASER Accuracy: 49. Result ID: 69d837cb47f34ce924c000a4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"PRIVATE":"YES","Backbone":"GPT-3.5-TURBO"}. This record does not assert independent reproduction. ### Phi-4 pass@N (Upper Bound) Accuracy: 0.991. Result ID: 69e9edd747f34ce924d19ddd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"# Sample":"-"}. This record does not assert independent reproduction. ### RLHFlow-PRM-Deepseek-Data Accuracy: 0.948. Result ID: 69e9edd747f34ce924d19de2. [Reporting paper]() Recorded evaluation configuration: {"# Sample":"253K"}. This record does not assert independent reproduction. ### RLHFlow-PRM-Mistral-Data Accuracy: 0.941. Result ID: 69e9edd747f34ce924d19de3. [Reporting paper]() Recorded evaluation configuration: {"# Sample":"273K"}. This record does not assert independent reproduction. ### Qwen2.5-Math-7B-PRM Accuracy: 0.941. Result ID: 69e9edd747f34ce924d19de6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"# Sample":"860K"}. This record does not assert independent reproduction. ### Qwen2.5-Math-7B-PRM800k Accuracy: 0.939. Result ID: 69e9edd747f34ce924d19de5. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"# Sample":"263K"}. This record does not assert independent reproduction. ### Majority Voting Accuracy: 0.932. Result ID: 69e9edd747f34ce924d19dde. [Reporting paper]() Recorded evaluation configuration: {"# Sample":"-"}. This record does not assert independent reproduction. ### Math-Shepherd-PRM-7B Accuracy: 0.932. Result ID: 69e9edd747f34ce924d19de1. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"# Sample":"440K"}. This record does not assert independent reproduction. ### Phi-4 pass@1 (Lower Bound) Accuracy: 0.93. Result ID: 69e9edd747f34ce924d19ddc. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"# Sample":"-"}. This record does not assert independent reproduction. ### ReProbe, Attn+Logit, Qwen3-8B-anno Accuracy: 0.93. Result ID: 69e9edd747f34ce924d19de7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"# Sample":"32K"}. This record does not assert independent reproduction. ### Skywork-PRM-1.5B Accuracy: 0.925. Result ID: 69e9edd747f34ce924d19ddf. [Reporting paper]() Recorded evaluation configuration: {"# Sample":"Unk"}. This record does not assert independent reproduction. ### Universal-PRM-Qwen2.5-Math-7B Accuracy: 0.925. Result ID: 69e9edd747f34ce924d19de4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"# Sample":"690K"}. This record does not assert independent reproduction. ### H4-Qwen2.5-PRM-1.5B-0.2 Accuracy: 0.918. Result ID: 69e9edd747f34ce924d19de0. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"# Sample":"369K"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.