# Let's Verify Step by Step

Source: [Let's Verify Step by Step](<https://www.sota2.com/research/paper/68fa6f9a1e209131cea468c7>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:52:28.284Z

Authors: Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe

Published: 2023-05-31T17:24:00.000Z

## Abstract

In recent years, large language models have greatly improved in their ability to perform complex multi-step reasoning. However, even state-of-the-art models still regularly produce logical mistakes. To train more reliable models, we can turn either to outcome supervision, which provides feedback for a final result, or process supervision, which provides feedback for each intermediate reasoning step. Given the importance of training reliable models, and given the high cost of human feedback, it is important to carefully compare the both methods. Recent work has already begun this comparison, but many questions still remain. We conduct our own investigation, finding that process supervision significantly outperforms outcome supervision for training models to solve problems from the challenging MATH dataset. Our process-supervised model solves 78% of problems from a representative subset of the MATH test set. Additionally, we show that active learning significantly improves the efficacy of process supervision. To support related research, we also release PRM800K, the complete dataset of 800,000 step-level human feedback labels used to train our best reward model.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [General Reasoning on StratQA](<https://www.sota2.com/research/sota/general-reasoning-on-stratqa.md>): General Reasoning; StratQA; Accuracy: 87.8.

- [Step-level correctness assessment on MATH (test)](<https://www.sota2.com/research/sota/step-level-correctness-assessment-on-math-test.md>): Step-level correctness assessment; MATH (test); PR-AUC: 58.6.

- [Step-level reasoning verification on Trips](<https://www.sota2.com/research/sota/step-level-reasoning-verification-on-trips.md>): Step-level reasoning verification; Trips; PR-AUC: 82.5.

- [Mathematical Reasoning on MATH500 (BCE, MSE, Q-value rankings)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math500-bce-mse-q-value-rankings.md>): Mathematical Reasoning; MATH500; BCE: 18.48.

- [Step-level reasoning verification on Aggregate ID/OOD](<https://www.sota2.com/research/sota/step-level-reasoning-verification-on-aggregate-id-ood.md>): Step-level reasoning verification; Aggregate ID/OOD; ID Score: 37.3.

- [Logical Reasoning on LogiQA (Accuracy %)](<https://www.sota2.com/research/sota/logical-reasoning-on-logiqa-accuracy-percent.md>): Logical reasoning; LogiQA; LogiQA Accuracy: 78.6.

- [Code Generation on HumanE](<https://www.sota2.com/research/sota/code-generation-on-humane.md>): Code Generation; HumanE; Accuracy: 84.8.

- [Logical Reasoning on ProofW (Accuracy)](<https://www.sota2.com/research/sota/logical-reasoning-on-proofw-accuracy.md>): Logical reasoning; ProofW; Accuracy: 83.6.

- [Code Reasoning on HumanE](<https://www.sota2.com/research/sota/code-reasoning-on-humane.md>): Code Reasoning; HumanE; Accuracy: 84.6.

- [Mathematical Reasoning on GSM8K (Accuracy @ N Tokens)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gsm8k-accuracy-at-n-tokens.md>): Mathematical Reasoning; GSM8K; Accuracy @ 128 Tokens: 92.7.

- [Mathematical Reasoning on MATH-500 (Context Length Accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-500-context-length-accuracy.md>): Mathematical Reasoning; MATH 500; Accuracy @ 128: 60.3.

- [Step-level Reasoning Verification on StrQA](<https://www.sota2.com/research/sota/step-level-reasoning-verification-on-strqa.md>): Step-level reasoning verification; StrQA; PR-AUC: 35.5.

- [Question Answering on HalluQA](<https://www.sota2.com/research/sota/question-answering-on-halluqa.md>): Question Answering; HalluQA; Accuracy: 35.44.

- [Question Answering on ChineseFactEval](<https://www.sota2.com/research/sota/question-answering-on-chinesefacteval.md>): Question Answering; ChineseFactEval; Accuracy: 64.

- [Logical Reasoning on ProofW (Accuracy, Token Cost)](<https://www.sota2.com/research/sota/logical-reasoning-on-proofw-accuracy-token-cost.md>): Logical reasoning; ProofW; Accuracy: 76.8.

- [Mathematical Reasoning on MATH-Vision mini (test)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-vision-mini-test.md>): Mathematical Reasoning; MATH-Vision mini (test); ALG: 31.58.

- [Degree of Change on PerturbQA Jurkat](<https://www.sota2.com/research/sota/degree-of-change-on-perturbqa-jurkat.md>): Degree of Change; PerturbQA Jurkat; Accuracy: 61.77.

- [Degree of Change on PerturbQA RPE1](<https://www.sota2.com/research/sota/degree-of-change-on-perturbqa-rpe1.md>): Degree of Change; PerturbQA RPE1; Accuracy: 72.58.

- [Differential Expression on PerturbQA HepG2](<https://www.sota2.com/research/sota/differential-expression-on-perturbqa-hepg2.md>): Differential Expression; PerturbQA (HepG2); Accuracy: 47.27.

- [Differential Expression on PerturbQA Jurkat](<https://www.sota2.com/research/sota/differential-expression-on-perturbqa-jurkat.md>): Differential Expression; PerturbQA Jurkat; Accuracy: 48.71.

- [Differential Expression on PerturbQA K562](<https://www.sota2.com/research/sota/differential-expression-on-perturbqa-k562.md>): Differential Expression; PerturbQA K562; Accuracy: 44.48.

- [Differential Expression on PerturbQA RPE1](<https://www.sota2.com/research/sota/differential-expression-on-perturbqa-rpe1.md>): Differential Expression; PerturbQA RPE1; Accuracy: 45.2.

- [Mathematical Reasoning on ProofNet (test)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-proofnet-test.md>): Mathematical Reasoning; ProofNet (test); Accuracy: 95.2.

- [Planning on Calendar (test)](<https://www.sota2.com/research/sota/planning-on-calendar-test.md>): Planning; Calendar (test); Accuracy: 45.

- [Logical Reasoning on LogiQA (Accuracy)](<https://www.sota2.com/research/sota/logical-reasoning-on-logiqa-accuracy.md>): Logical reasoning; LogiQA; Accuracy: 78.9.

- [Mathematical Reasoning on GK 2023](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gk-2023.md>): Mathematical Reasoning; GK 2023; Accuracy: 48.3.

- [Step-level correctness assessment on ProofNet (test)](<https://www.sota2.com/research/sota/step-level-correctness-assessment-on-proofnet-test.md>): Step-level correctness assessment; ProofNet (test); PR-AUC: 30.1.

- [Step-level Reasoning Verification on GSM8k](<https://www.sota2.com/research/sota/step-level-reasoning-verification-on-gsm8k.md>): Step-level reasoning verification; GSM8K; PR-AUC: 89.4.

- [Step-level reasoning verification on SciQA](<https://www.sota2.com/research/sota/step-level-reasoning-verification-on-sciqa.md>): Step-level reasoning verification; SciQA; PR-AUC: 35.

- [General Reasoning on BBH (SR %)](<https://www.sota2.com/research/sota/general-reasoning-on-bbh-sr-percent.md>): General Reasoning; BBH; Success Rate (BBH General Reasoning): 80.4.

- [Logic Consistency Evaluation on Aggregate of 16 tasks](<https://www.sota2.com/research/sota/logic-consistency-evaluation-on-aggregate-of-16-tasks.md>): Logic Consistency Evaluation; Aggregate of 16 tasks; LCS (Avg): 79.8.

- [Question Answering on HaluEval QA](<https://www.sota2.com/research/sota/question-answering-on-halueval-qa.md>): Question Answering; HaluEval QA; Accuracy: 39.

- [Logical Reasoning on LogiQA (Accuracy, Token Cost)](<https://www.sota2.com/research/sota/logical-reasoning-on-logiqa-accuracy-token-cost.md>): Logical reasoning; LogiQA; Accuracy: 72.3.

- [Step-level Reasoning Verification on Step-level Verification Suite Math, Planning, QA](<https://www.sota2.com/research/sota/step-level-reasoning-verification-on-step-level-verification-suite-math-planning-qa.md>): Detecting incorrect reasoning steps; Step-level Verification Suite Math, Planning, QA; MATH Accuracy: 0.217.

- [Mathematical Reasoning on AMC23 (Pass@1 accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-amc23-pass-at-1-accuracy.md>): Mathematical Reasoning; AMC23; Pass@1 Accuracy: 60.

- [Degree of Change on PerturbQA HepG2](<https://www.sota2.com/research/sota/degree-of-change-on-perturbqa-hepg2.md>): Degree of Change; PerturbQA (HepG2); Accuracy: 73.53.

- [Degree of Change on PerturbQA K562](<https://www.sota2.com/research/sota/degree-of-change-on-perturbqa-k562.md>): Degree of Change; PerturbQA K562; Accuracy: 84.87.

- [Mathematical Reasoning on MathVision (Acc., Tokens)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-mathvision-acc-tokens.md>): Mathematical Reasoning; MathVision; Accuracy: 25.

- [Information Visual Question Answering on InfoVQA (test)](<https://www.sota2.com/research/sota/information-visual-question-answering-on-infovqa-test.md>): Information Visual Question Answering; InfoVQA (test); ANLS: 86.2.

- [Step-level Reasoning Verification on ProofNet](<https://www.sota2.com/research/sota/step-level-reasoning-verification-on-proofnet.md>): Step-level reasoning verification; ProofNet; PR-AUC: 45.4.

- [Step-level reasoning verification on Calendar](<https://www.sota2.com/research/sota/step-level-reasoning-verification-on-calendar.md>): Step-level reasoning verification; Calendar; PR-AUC: 59.4.

- [Reasoning on SAT](<https://www.sota2.com/research/sota/reasoning-on-sat.md>): Reasoning; SAT; Accuracy (SAT): 95.2.

- [Visual Reasoning on CharXiv (val)](<https://www.sota2.com/research/sota/visual-reasoning-on-charxiv-val.md>): Visual Reasoning; CharXiv (val); Text in Chart Accuracy: 34.09.

- [Mathematical Reasoning on MATH Overall](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-overall.md>): Mathematical Reasoning; MATH Overall; SR: 58.4.

- [Mathematical Reasoning on AIME 24-26 (Pass@1 accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-24-26-pass-at-1-accuracy.md>): Mathematical Reasoning; AIME 24-26; Pass@1 Accuracy: 14.44.

- [Planning on Trips (test)](<https://www.sota2.com/research/sota/planning-on-trips-test.md>): Planning; Trips (test); Accuracy: 13.1.

- [Step-level verification on MATH](<https://www.sota2.com/research/sota/step-level-verification-on-math.md>): Step-level verification; MATH; PR-AUC: 40.9.

- [Mathematical Reasoning on MATH 500 (Pass@k and BoN)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-500-pass-at-k-and-bon.md>): Mathematical Reasoning; MATH 500; Pass@1: 57.7.

- [Science Question Answering on ScienceQA](<https://www.sota2.com/research/sota/science-question-answering-on-scienceqa.md>): Science Question Answering; ScienceQA; Accuracy: 97.5.

- [Mathematical Reasoning on GSM8k (Accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gsm8k-accuracy.md>): Mathematical Reasoning; GSM8K; Accuracy: 100.

- [Question Answering on StrategyQA (test)](<https://www.sota2.com/research/sota/question-answering-on-strategyqa-test.md>): Question Answering; StrategyQA (test); Task Accuracy: 86.7.

- [Code Reasoning on MBPP](<https://www.sota2.com/research/sota/code-reasoning-on-mbpp.md>): Code Reasoning; MBPP; MBPP Execution Accuracy: 73.2.

- [Step-level correctness assessment on Meetings (test)](<https://www.sota2.com/research/sota/step-level-correctness-assessment-on-meetings-test.md>): Step-level correctness assessment; Meetings (test); PR-AUC: 76.8.

- [Step-level correctness assessment on StrQA (test)](<https://www.sota2.com/research/sota/step-level-correctness-assessment-on-strqa-test.md>): Step-level correctness assessment; StrQA (test); PR-AUC: 36.2.

- [Step-level reasoning verification on Meetings](<https://www.sota2.com/research/sota/step-level-reasoning-verification-on-meetings.md>): Step-level reasoning verification; Meetings; PR-AUC: 59.9.

- [Multimodal Reasoning on V\* Bench Tool-needed](<https://www.sota2.com/research/sota/multimodal-reasoning-on-v-bench-tool-needed.md>): Multimodal Reasoning; V\* Bench Tool-needed; Accuracy: 88.9.

- [Mathematical Reasoning on MATH Level-4](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-level-4.md>): Mathematical Reasoning; MATH Level-4; SR (%): 62.8.

- [General Reasoning on MMLU](<https://www.sota2.com/research/sota/general-reasoning-on-mmlu.md>): General Reasoning; MMLU; MMLU Accuracy: 89.

- [Multimodal Understanding on MMBench EN v1.1](<https://www.sota2.com/research/sota/multimodal-understanding-on-mmbench-en-v1-1.md>): Multimodal Understanding; MMBench EN v1.1; Accuracy: 87.8.

- [Code Reasoning on MBPP (Accuracy, Token Cost)](<https://www.sota2.com/research/sota/code-reasoning-on-mbpp-accuracy-token-cost.md>): Code Reasoning; MBPP; Accuracy: 65.8.

- [Step-level correctness assessment on GSM8k (test)](<https://www.sota2.com/research/sota/step-level-correctness-assessment-on-gsm8k-test.md>): Step-level correctness assessment; GSM8K (test); PR-AUC: 61.3.

- [Step-level correctness assessment on Calendar (test)](<https://www.sota2.com/research/sota/step-level-correctness-assessment-on-calendar-test.md>): Step-level correctness assessment; Calendar (test); PR-AUC: 0.727.

- [Mathematical Reasoning on PRM800K (test)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-prm800k-test.md>): Mathematical Reasoning; PRM800K (test); Accuracy: 70.

- [Graph-structured planning on Graph-structured planning pooled over four benchmarks (test)](<https://www.sota2.com/research/sota/graph-structured-planning-on-graph-structured-planning-pooled-over-four-benchmarks-test.md>): Graph-structured planning; Graph-structured planning pooled over four benchmarks (test); Success Rate (SR): 68.4.

- [Multimodal Mathematical Reasoning on MATH-Vision (testmini)](<https://www.sota2.com/research/sota/multimodal-mathematical-reasoning-on-math-vision-testmini.md>): Multimodal Mathematical Reasoning; MATH-Vision (testmini); Alg Score: 5.26.

- [Process-level Reward Modeling on PROCESSBENCH MATH](<https://www.sota2.com/research/sota/process-level-reward-modeling-on-processbench-math.md>): Process-level Reward Modeling; PROCESSBENCH MATH; Error Rate: 48.

- [Process-level Reward Modeling on PROCESSBENCH Olymp.Bench](<https://www.sota2.com/research/sota/process-level-reward-modeling-on-processbench-olymp-bench.md>): Process-level Reward Modeling; PROCESSBENCH Olymp.Bench; Error: 35.7.

- [Process-level Reward Modeling on PROCESSBENCH Omni-MATH](<https://www.sota2.com/research/sota/process-level-reward-modeling-on-processbench-omni-math.md>): Process-level Reward Modeling; PROCESSBENCH Omni-MATH; Error Rate: 29.8.

- [Step-level correctness assessment on Trips (test)](<https://www.sota2.com/research/sota/step-level-correctness-assessment-on-trips-test.md>): Step-level correctness assessment; Trips (test); PR-AUC: 70.8.

- [Mathematical Reasoning on ProofNet](<https://www.sota2.com/research/sota/mathematical-reasoning-on-proofnet.md>): Mathematical Reasoning; ProofNet; Accuracy: 92.4.

- [Video Multimodal Understanding on VideoMMMU](<https://www.sota2.com/research/sota/video-multimodal-understanding-on-videommmu.md>): Video Multimodal Understanding; VideoMMMU; Accuracy: 67.6.

- [Step-level correctness assessment on SciQA (test)](<https://www.sota2.com/research/sota/step-level-correctness-assessment-on-sciqa-test.md>): Step-level correctness assessment; SciQA (test); PR-AUC: 40.4.

- [Mathematical Reasoning on MATH level 5](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-level-5.md>): Mathematical Reasoning; MATH level 5; Success Rate: 51.9.

- [Mathematical Reasoning on GPQA-D](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gpqa-d.md>): Mathematical Reasoning; GPQA D; Accuracy: 72.

- [Specification Alignment on Specification Alignment Evaluation Set (full (1500))](<https://www.sota2.com/research/sota/specification-alignment-on-specification-alignment-evaluation-set-full-1500.md>): Specification Alignment; Specification Alignment Evaluation Set (full (1500)); Safety Score: 64.2.

- [Mathematical Reasoning on Math ID GSM8k ProofNet](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-id-gsm8k-proofnet.md>): Mathematical Reasoning; Math ID GSM8k ProofNet; GSM8k Accuracy: 97.3.

- [General Reasoning on Aggregate Reasoning Suite ID & OOD](<https://www.sota2.com/research/sota/general-reasoning-on-aggregate-reasoning-suite-id-and-ood.md>): General Reasoning; Aggregate Reasoning Suite ID & OOD; ID Average Score: 88.1.

- [Math Reasoning on GSM8K (test)](<https://www.sota2.com/research/sota/math-reasoning-on-gsm8k-test.md>): Math Reasoning; GSM8K (test); Accuracy: 92.6.

- [Step Verification Efficiency Evaluation on MATH (test)](<https://www.sota2.com/research/sota/step-verification-efficiency-evaluation-on-math-test.md>): Step Verification Efficiency Evaluation; MATH (test); Runtime (s): 37.

- [Question Answering on ChineseSimpleQA](<https://www.sota2.com/research/sota/question-answering-on-chinesesimpleqa.md>): Question Answering; ChineseSimpleQA; Accuracy: 35.

- [Visual Spatial Intelligence on VSI-Bench](<https://www.sota2.com/research/sota/visual-spatial-intelligence-on-vsi-bench.md>): Visual Spatial Intelligence; VSI-Bench; Average Score: 62.3.

- [General Reasoning on MMLU (Accuracy and Token Cost)](<https://www.sota2.com/research/sota/general-reasoning-on-mmlu-accuracy-and-token-cost.md>): General Reasoning; MMLU; Accuracy: 80.1.

- [Question Answering on QA OOD StrQA SciQA](<https://www.sota2.com/research/sota/question-answering-on-qa-ood-strqa-sciqa.md>): Question Answering; QA OOD StrQA SciQA; StrQA Accuracy: 87.1.

- [General Knowledge on MMLU (Accuracy)](<https://www.sota2.com/research/sota/general-knowledge-on-mmlu-accuracy.md>): General Knowledge; MMLU; MMLU General Knowledge Accuracy: 87.7.

- [Mathematical Reasoning on SVAMP](<https://www.sota2.com/research/sota/mathematical-reasoning-on-svamp.md>): Mathematical Reasoning; SVAMP; Accuracy: 93.4.

- [Process Reward Model Assessment on PROCESSBENCH](<https://www.sota2.com/research/sota/process-reward-model-assessment-on-processbench.md>): Process Reward Model Assessment; PROCESSBENCH; GSM8K Accuracy: 49.1.

- [Planning on Planning OOD Trips Meetings Calendar](<https://www.sota2.com/research/sota/planning-on-planning-ood-trips-meetings-calendar.md>): Planning; Planning OOD Trips Meetings Calendar; Trips Planned Count: 5.9.

- [Code Generation on MBPP (Acc. %, RSR %)](<https://www.sota2.com/research/sota/code-generation-on-mbpp-acc-percent-rsr-percent.md>): Code Generation; MBPP; Accuracy: 73.8.

- [Math Reasoning on OlympiadB](<https://www.sota2.com/research/sota/math-reasoning-on-olympiadb.md>): Math Reasoning; OlympiadB; Accuracy: 83.5.

- [Reasoning Question Answering on StrategyQA (ACC & AUC)](<https://www.sota2.com/research/sota/reasoning-question-answering-on-strategyqa-acc-and-auc.md>): Reasoning Question Answering; StrategyQA; Accuracy: 0.939.

- [Multitask Language Understanding on MMLU (Accuracy)](<https://www.sota2.com/research/sota/multitask-language-understanding-on-mmlu-accuracy.md>): Multitask Language Understanding; MMLU; Accuracy: 82.6.

- [Accuracy on AMC Math Reasoning](<https://www.sota2.com/research/sota/accuracy-on-amc-math-reasoning.md>): Math Reasoning; AMC; Accuracy: 61.

- [Mathematical Reasoning on Minerva Math (Pass@1 accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-minerva-math-pass-at-1-accuracy.md>): Mathematical Reasoning; Minerva Math; Pass@1 Accuracy: 25.37.

- [Math Reasoning on JEEBench](<https://www.sota2.com/research/sota/math-reasoning-on-jeebench.md>): Math Reasoning; JEEBench; Accuracy: 63.5.

- [Science Question Answering on ScienceQA (test)](<https://www.sota2.com/research/sota/science-question-answering-on-scienceqa-test.md>): Science Question Answering; ScienceQA (test); Average Accuracy: 91.2.

- [Mathematical Reasoning on AIME 25 (accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-25-accuracy.md>): Mathematical Reasoning; AIME 25; Accuracy: 86.

- [Math Reasoning on MATH](<https://www.sota2.com/research/sota/math-reasoning-on-math.md>): Math Reasoning; MATH; Accuracy: 56.9.

- [Math Reasoning on MATH500 (Accuracy)](<https://www.sota2.com/research/sota/math-reasoning-on-math500-accuracy.md>): Math Reasoning; MATH500; Accuracy: 85.

- [Multimodal Mathematical Reasoning on MathVista mini (test)](<https://www.sota2.com/research/sota/multimodal-mathematical-reasoning-on-mathvista-mini-test.md>): Multimodal Mathematical Reasoning; MathVista mini (test); Overall Accuracy: 62.

- [Mathematical Reasoning on HMMT 2025](<https://www.sota2.com/research/sota/mathematical-reasoning-on-hmmt-2025.md>): Mathematical Reasoning; HMMT 2025; Accuracy: 70.05.

- [Mathematical Reasoning on GSM8K](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gsm8k.md>): Mathematical Reasoning; GSM8K; Accuracy: 93.3.

- [Visual Mathematical Reasoning on MathVista (testmini)](<https://www.sota2.com/research/sota/visual-mathematical-reasoning-on-mathvista-testmini.md>): Visual Mathematical Reasoning; MathVista (testmini); Accuracy: 57.7.

- [Grade School Math Reasoning on GSM8K (Accuracy)](<https://www.sota2.com/research/sota/grade-school-math-reasoning-on-gsm8k-accuracy.md>): Grade School Math Reasoning; GSM8K; Accuracy (GSM8K): 90.5.

- [Question Answering on TruthfulQA](<https://www.sota2.com/research/sota/question-answering-on-truthfulqa.md>): Question Answering; TruthfulQA; Accuracy: 43.5.

- [Mathematical Problem Solving on MATH](<https://www.sota2.com/research/sota/mathematical-problem-solving-on-math.md>): Mathematical Problem Solving; MATH; Accuracy: 50.6.

- [Mathematical Reasoning on GSM8K (test)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gsm8k-test.md>): Mathematical Reasoning; GSM8K (test); Accuracy: 89.2.

- [Question Answering on SimpleQA](<https://www.sota2.com/research/sota/question-answering-on-simpleqa.md>): Question Answering; SimpleQA; Accuracy: 6.

- [Mathematical Reasoning on MathVerse](<https://www.sota2.com/research/sota/mathematical-reasoning-on-mathverse.md>): Mathematical Reasoning; MathVerse; Accuracy: 45.74.

- [Mathematical Reasoning on OlympiadBench (Accuracy, Tokens)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-olympiadbench-accuracy-tokens.md>): Mathematical Reasoning; OlympiadBench; Accuracy: 16.67.

- [Mathematical Reasoning on GSM8K (Accuracy & Token Count)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gsm8k-accuracy-and-token-count.md>): Mathematical Reasoning; GSM8K; Accuracy: 86.5.

- [Mathematical Reasoning on MATH](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math.md>): Mathematical Reasoning; MATH; Accuracy: 57.8.

- [Mathematical Reasoning on AIME 2024 (AIME24, Math Avg)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-aime-2024-aime24-math-avg.md>): Mathematical Reasoning; AIME 2024; Accuracy: 33.3.

- [Mathematical Reasoning on MathVista (Accuracy)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-mathvista-accuracy.md>): Mathematical Reasoning; MathVista; Accuracy: 63.3.

- [Mathematical Reasoning on MATH 500 (Acc, SL, SE)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math-500-acc-sl-se.md>): Mathematical Reasoning; MATH 500; Accuracy (Acc): 76.1.

- [Mathematical Reasoning on MATH500 (test)](<https://www.sota2.com/research/sota/mathematical-reasoning-on-math500-test.md>): Mathematical Reasoning; MATH500 (test); Accuracy: 62.2.

- [Text-to-image generation on GenEval (Detailed Compositional Metrics)](<https://www.sota2.com/research/sota/text-to-image-generation-on-geneval-detailed-compositional-metrics.md>): Text-to-Image Generation; GenEval; Overall Score: 44.

- [Multi-modal Video Understanding on MVBench](<https://www.sota2.com/research/sota/multi-modal-video-understanding-on-mvbench.md>): Multi-modal Video Understanding; MVBench; Score: not recorded.

- [Reasoning on MATH (AUROC/FPR95)](<https://www.sota2.com/research/sota/reasoning-on-math-auroc-fpr95.md>): Reasoning; MATH; AUROC: not recorded.

- [Mathematical Reasoning on Gaokao 2023](<https://www.sota2.com/research/sota/mathematical-reasoning-on-gaokao-2023.md>): Mathematical Reasoning; Gaokao 2023; Accuracy: not recorded.

- [Text-to-Image Generation on TIIF-Bench](<https://www.sota2.com/research/sota/text-to-image-generation-on-tiif-bench.md>): Text-to-Image Generation; TIIF-Bench; Basic Short Score: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.20050>)

[PDF](<https://arxiv.org/pdf/2305.20050.pdf>)

[Implementation](<https://github.com/openai/prm800k>)
