# Logical Reasoning on LogiQA (Pass@1, Variance) Source: [Logical Reasoning on LogiQA (Pass@1, Variance)]() Collection index: [llms.txt]() Data updated: 2026-02-26T05:57:34.171Z Evaluation of model performance on logical reasoning tasks using the LogiQA dataset, measured by Pass@1 accuracy and result variance. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Logical Reasoning | | Dataset (including any stated split) | LogiQA | | Metric | Pass@1 Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 18 of 18 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Pass@1 Accuracy](): Higher is better; unit: PERCENT. - [Variance](): Lower is better; unit: SCALAR. ## Recorded results | Method | Pass@1 Accuracy | Result date | Report paper | | --- | --- | --- | --- | | SFT w/ Short CoT | 0.88 | Not recorded | 68fa79621e209131cea7d54d | | DeepSeek-R1 | 0.85 | Not recorded | 68fa79621e209131cea7d54d | | RULEREASONER-8B | 0.835 | Not recorded | 68fa79621e209131cea7d54d | | OpenAI o1 | 0.821 | Not recorded | 68fa79621e209131cea7d54d | | Claude-3.7-Sonnet | 0.815 | Not recorded | 68fa79621e209131cea7d54d | | Chain-of-Logic | 0.806 | Not recorded | 68fa79621e209131cea7d54d | | ADARFT | 0.805 | Not recorded | 68fa79621e209131cea7d54d | | RULEREASONER-4B | 0.805 | Not recorded | 68fa79621e209131cea7d54d | | HtT | 0.791 | Not recorded | 68fa79621e209131cea7d54d | | RGFB | 0.791 | Not recorded | 68fa79621e209131cea7d54d | | SFT w/ Long CoT | 0.791 | Not recorded | 68fa79621e209131cea7d54d | | OpenAI o3-mini | 0.776 | Not recorded | 68fa79621e209131cea7d54d | | SFT w/o CoT | 0.761 | Not recorded | 68fa79621e209131cea7d54d | | DAPO | 0.746 | Not recorded | 68fa79621e209131cea7d54d | | Easy-to-hard RL | 0.746 | Not recorded | 68fa79621e209131cea7d54d | | Data-balance RL | 0.731 | Not recorded | 68fa79621e209131cea7d54d | | GRPO | 0.701 | Not recorded | 68fa79621e209131cea7d54d | | Dr. GRPO | 0.656 | Not recorded | 68fa79621e209131cea7d54d | ### SFT w/ Short CoT Pass@1 Accuracy: 0.88. Result ID: 699a872f5698665ec4113926. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"BEHAVIORAL CLONING","Reasoning Strategy":"Short CoT"}. This record does not assert independent reproduction. ### DeepSeek-R1 Pass@1 Accuracy: 0.85. Result ID: 699a872f5698665ec4113924. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"FRONTIER REASONERS"}. This record does not assert independent reproduction. ### RULEREASONER-8B Pass@1 Accuracy: 0.835. Result ID: 699a872f5698665ec411392f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"RULEREASONER (Ours)","Model Scale":"8B"}. This record does not assert independent reproduction. ### OpenAI o1 Pass@1 Accuracy: 0.821. Result ID: 699a872f5698665ec4113921. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"FRONTIER REASONERS"}. This record does not assert independent reproduction. ### Claude-3.7-Sonnet Pass@1 Accuracy: 0.815. Result ID: 699a872f5698665ec4113923. [Reporting paper]() Recorded evaluation configuration: {"Method Category":"FRONTIER REASONERS"}. This record does not assert independent reproduction. ### Chain-of-Logic Pass@1 Accuracy: 0.806. Result ID: 699a872f5698665ec4113920. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"PRIOR RBRS"}. This record does not assert independent reproduction. ### ADARFT Pass@1 Accuracy: 0.805. Result ID: 699a872f5698665ec411392d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"CURRICULUM LEARNING"}. This record does not assert independent reproduction. ### RULEREASONER-4B Pass@1 Accuracy: 0.805. Result ID: 699a872f5698665ec411392e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"RULEREASONER (Ours)","Model Scale":"4B"}. This record does not assert independent reproduction. ### HtT Pass@1 Accuracy: 0.791. Result ID: 699a872f5698665ec411391e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"PRIOR RBRS"}. This record does not assert independent reproduction. ### RGFB Pass@1 Accuracy: 0.791. Result ID: 699a872f5698665ec411391f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"PRIOR RBRS"}. This record does not assert independent reproduction. ### SFT w/ Long CoT Pass@1 Accuracy: 0.791. Result ID: 699a872f5698665ec4113927. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"BEHAVIORAL CLONING","Reasoning Strategy":"Long CoT"}. This record does not assert independent reproduction. ### OpenAI o3-mini Pass@1 Accuracy: 0.776. Result ID: 699a872f5698665ec4113922. [Reporting paper]() Recorded evaluation configuration: {"Method Category":"FRONTIER REASONERS"}. This record does not assert independent reproduction. ### SFT w/o CoT Pass@1 Accuracy: 0.761. Result ID: 699a872f5698665ec4113925. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Method Category":"BEHAVIORAL CLONING","Reasoning Strategy":"w/o CoT"}. This record does not assert independent reproduction. ### DAPO Pass@1 Accuracy: 0.746. Result ID: 699a872f5698665ec411392a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"ADVANCED RLVRS"}. This record does not assert independent reproduction. ### Easy-to-hard RL Pass@1 Accuracy: 0.746. Result ID: 699a872f5698665ec411392c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"CURRICULUM LEARNING"}. This record does not assert independent reproduction. ### Data-balance RL Pass@1 Accuracy: 0.731. Result ID: 699a872f5698665ec411392b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"CURRICULUM LEARNING"}. This record does not assert independent reproduction. ### GRPO Pass@1 Accuracy: 0.701. Result ID: 699a872f5698665ec4113928. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Method Category":"ADVANCED RLVRS"}. This record does not assert independent reproduction. ### Dr. GRPO Pass@1 Accuracy: 0.656. Result ID: 699a872f5698665ec4113929. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Method Category":"ADVANCED RLVRS"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.