# Physical Reasoning on PHYRE DeepPHY (test) Source: [Physical Reasoning on PHYRE DeepPHY (test)]() Collection index: [llms.txt]() Data updated: 2026-03-17T07:30:03.186Z Evaluation of model performance on the Physical Reasoning task using the PHYRE DeepPHY test split. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Physical Reasoning | | Dataset (including any stated split) | PHYRE DeepPHY (test) | | Metric | Att. 1 Score | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 36 of 36 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Att. 1 Score](): Higher is better; unit: PERCENT. - [Att. 10 Score](): Higher is better; unit: PERCENT. ## Recorded results | Method | Att. 1 Score | Result date | Report paper | | --- | --- | --- | --- | | Qwen-7B^P&I GRPO | 14.92 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P&I GRPO | 14.63 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P&I GRPO | 14.11 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P&I GRPO | 14 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P GRPO | 13.18 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P GRPO | 13 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P&I SFT | 10.7 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P&I SFT | 10.42 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P GRPO | 9.85 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P GRPO | 9.5 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P&I SFT | 7.95 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P&I SFT | 7.5 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P SFT | 3.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P SFT | 3.4 | Not recorded | 69b8e0060636e5bf4afbe99b | | GPT-o3 | 3.03 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-72B | 2.43 | Not recorded | 69b8e0060636e5bf4afbe99b | | Gemini-2.5-Pro | 2.17 | Not recorded | 69b8e0060636e5bf4afbe99b | | Gemini-2.5-Pro | 2.17 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-72B | 1.78 | Not recorded | 69b8e0060636e5bf4afbe99b | | Claude 4.0 Opus | 1.73 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P SFT | 1.52 | Not recorded | 69b8e0060636e5bf4afbe99b | | MOCK | 1.5 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P SFT | 1.33 | Not recorded | 69b8e0060636e5bf4afbe99b | | Claude 4.0 Opus | 1.11 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^I SFT | 0.82 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^I SFT | 0.74 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B | 0.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^I SFT | 0.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^I GRPO | 0.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^I GRPO | 0.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^I GRPO | 0.66 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^I SFT | 0.33 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B | 0.17 | Not recorded | 69b8e0060636e5bf4afbe99b | | GPT-o3 | 0.17 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^I GRPO | 0.17 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-32B | 0.03 | Not recorded | 69b8e0060636e5bf4afbe99b | ### Qwen-7B^P&I GRPO Att. 1 Score: 14.92. Result ID: 69b902da47f34ce924a12268. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P&I GRPO","World Model":"Qwen-7B^P&I"}. This record does not assert independent reproduction. ### Qwen-7B^P&I GRPO Att. 1 Score: 14.63. Result ID: 69b902da47f34ce924a12267. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P&I GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P&I GRPO Att. 1 Score: 14.11. Result ID: 69b902da47f34ce924a1225c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P&I GRPO","World Model":"Qwen-3B^P&I"}. This record does not assert independent reproduction. ### Qwen-3B^P&I GRPO Att. 1 Score: 14. Result ID: 69b902da47f34ce924a1225b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P&I GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^P GRPO Att. 1 Score: 13.18. Result ID: 69b902da47f34ce924a12264. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P GRPO","World Model":"Qwen-7B^P"}. This record does not assert independent reproduction. ### Qwen-7B^P GRPO Att. 1 Score: 13. Result ID: 69b902da47f34ce924a12263. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P&I SFT Att. 1 Score: 10.7. Result ID: 69b902da47f34ce924a12256. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P&I SFT","World Model":"Qwen-3B^P&I"}. This record does not assert independent reproduction. ### Qwen-3B^P&I SFT Att. 1 Score: 10.42. Result ID: 69b902da47f34ce924a12255. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P&I SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P GRPO Att. 1 Score: 9.85. Result ID: 69b902da47f34ce924a12258. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P GRPO","World Model":"Qwen-3B^P"}. This record does not assert independent reproduction. ### Qwen-3B^P GRPO Att. 1 Score: 9.5. Result ID: 69b902da47f34ce924a12257. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^P&I SFT Att. 1 Score: 7.95. Result ID: 69b902da47f34ce924a12262. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P&I SFT","World Model":"Qwen-7B^P&I"}. This record does not assert independent reproduction. ### Qwen-7B^P&I SFT Att. 1 Score: 7.5. Result ID: 69b902da47f34ce924a12261. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P&I SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^P SFT Att. 1 Score: 3.67. Result ID: 69b902da47f34ce924a1225d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^P SFT Att. 1 Score: 3.4. Result ID: 69b902da47f34ce924a1225e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P SFT","World Model":"Qwen-7B^P"}. This record does not assert independent reproduction. ### GPT-o3 Att. 1 Score: 3.03. Result ID: 69b902da47f34ce924a1224f. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"GPT-o3","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-72B Att. 1 Score: 2.43. Result ID: 69b902da47f34ce924a12249. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-72B","World Model":"-"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro Att. 1 Score: 2.17. Result ID: 69b902da47f34ce924a1224d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Gemini-2.5-Pro","World Model":"-"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro Att. 1 Score: 2.17. Result ID: 69b902da47f34ce924a1224e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Gemini-2.5-Pro","World Model":"Gemini-2.5-Pro"}. This record does not assert independent reproduction. ### Qwen-72B Att. 1 Score: 1.78. Result ID: 69b902da47f34ce924a1224a. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-72B","World Model":"Qwen-72B"}. This record does not assert independent reproduction. ### Claude 4.0 Opus Att. 1 Score: 1.73. Result ID: 69b902da47f34ce924a1224b. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Claude 4.0 Opus","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P SFT Att. 1 Score: 1.52. Result ID: 69b902da47f34ce924a12252. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P SFT","World Model":"Qwen-3B^P"}. This record does not assert independent reproduction. ### MOCK Att. 1 Score: 1.5. Result ID: 69b902da47f34ce924a12245. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"MOCK","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P SFT Att. 1 Score: 1.33. Result ID: 69b902da47f34ce924a12251. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P SFT","World Model":"-"}. This record does not assert independent reproduction. ### Claude 4.0 Opus Att. 1 Score: 1.11. Result ID: 69b902da47f34ce924a1224c. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Claude 4.0 Opus","World Model":"Claude 4.0 Opus"}. This record does not assert independent reproduction. ### Qwen-7B^I SFT Att. 1 Score: 0.82. Result ID: 69b902da47f34ce924a12260. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^I SFT","World Model":"Qwen-7B^I"}. This record does not assert independent reproduction. ### Qwen-3B^I SFT Att. 1 Score: 0.74. Result ID: 69b902da47f34ce924a12254. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^I SFT","World Model":"Qwen-3B^I"}. This record does not assert independent reproduction. ### Qwen-7B Att. 1 Score: 0.67. Result ID: 69b902da47f34ce924a12247. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^I SFT Att. 1 Score: 0.67. Result ID: 69b902da47f34ce924a1225f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^I SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^I GRPO Att. 1 Score: 0.67. Result ID: 69b902da47f34ce924a12265. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^I GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^I GRPO Att. 1 Score: 0.67. Result ID: 69b902da47f34ce924a12266. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^I GRPO","World Model":"Qwen-7B^I"}. This record does not assert independent reproduction. ### Qwen-3B^I GRPO Att. 1 Score: 0.66. Result ID: 69b902da47f34ce924a1225a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^I GRPO","World Model":"Qwen-3B^I"}. This record does not assert independent reproduction. ### Qwen-3B^I SFT Att. 1 Score: 0.33. Result ID: 69b902da47f34ce924a12253. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^I SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B Att. 1 Score: 0.17. Result ID: 69b902da47f34ce924a12246. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B","World Model":"-"}. This record does not assert independent reproduction. ### GPT-o3 Att. 1 Score: 0.17. Result ID: 69b902da47f34ce924a12250. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"GPT-o3","World Model":"GPT-o3"}. This record does not assert independent reproduction. ### Qwen-3B^I GRPO Att. 1 Score: 0.17. Result ID: 69b902da47f34ce924a12259. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^I GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-32B Att. 1 Score: 0.03. Result ID: 69b902da47f34ce924a12248. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-32B","World Model":"-"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.