# Physical Reasoning on I-PHYRE DeepPHY (test) Source: [Physical Reasoning on I-PHYRE DeepPHY (test)]() Collection index: [llms.txt]() Data updated: 2026-03-17T07:30:03.221Z Evaluation of model performance on the Physical Reasoning task using the test split of the I-PHYRE DeepPHY dataset. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Physical Reasoning | | Dataset (including any stated split) | I-PHYRE DeepPHY (test) | | Metric | Attention Score 1 | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 36 of 36 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Attention Score 1](): Higher is better; unit: PERCENT. - [Attention Score 10](): Higher is better; unit: PERCENT. ## Recorded results | Method | Attention Score 1 | Result date | Report paper | | --- | --- | --- | --- | | Qwen-3B^I GRPO | 37.6 | Not recorded | 69b8e0060636e5bf4afbe99b | | Claude 4.0 Opus | 36.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^I GRPO | 36.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P&I GRPO | 36.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P&I GRPO | 35.66 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B | 32.42 | Not recorded | 69b8e0060636e5bf4afbe99b | | Claude 4.0 Opus | 30 | Not recorded | 69b8e0060636e5bf4afbe99b | | GPT-o3 | 30 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^I GRPO | 27.82 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^I GRPO | 26.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | GPT-o3 | 25 | Not recorded | 69b8e0060636e5bf4afbe99b | | MOCK | 23.33 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^I SFT | 23.33 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^I SFT | 23.21 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P&I SFT | 21.41 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P&I SFT | 21.29 | Not recorded | 69b8e0060636e5bf4afbe99b | | Gemini-2.5-Pro | 20 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P SFT | 20 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P&I SFT | 20 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P&I SFT | 20 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P SFT | 19.87 | Not recorded | 69b8e0060636e5bf4afbe99b | | Gemini-2.5-Pro | 16.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P GRPO | 16.67 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P GRPO | 15.68 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P SFT | 13.57 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B | 13.33 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-72B | 13.33 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P GRPO | 13.33 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P SFT | 13.33 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P&I GRPO | 13.33 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^P&I GRPO | 13.12 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-3B^P GRPO | 12.78 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^I SFT | 12.22 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-7B^I SFT | 11.34 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-72B | 10 | Not recorded | 69b8e0060636e5bf4afbe99b | | Qwen-32B | 0.17 | Not recorded | 69b8e0060636e5bf4afbe99b | ### Qwen-3B^I GRPO Attention Score 1: 37.6. Result ID: 69b902da47f34ce924a1227e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^I GRPO","World Model":"Qwen-3B^I"}. This record does not assert independent reproduction. ### Claude 4.0 Opus Attention Score 1: 36.67. Result ID: 69b902da47f34ce924a1226f. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Claude 4.0 Opus","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^I GRPO Attention Score 1: 36.67. Result ID: 69b902da47f34ce924a1227d. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^I GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P&I GRPO Attention Score 1: 36.67. Result ID: 69b902da47f34ce924a1227f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P&I GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P&I GRPO Attention Score 1: 35.66. Result ID: 69b902da47f34ce924a12280. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P&I GRPO","World Model":"Qwen-3B^P&I"}. This record does not assert independent reproduction. ### Qwen-7B Attention Score 1: 32.42. Result ID: 69b902da47f34ce924a1226b. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B","World Model":"-"}. This record does not assert independent reproduction. ### Claude 4.0 Opus Attention Score 1: 30. Result ID: 69b902da47f34ce924a12270. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Claude 4.0 Opus","World Model":"Claude 4.0 Opus"}. This record does not assert independent reproduction. ### GPT-o3 Attention Score 1: 30. Result ID: 69b902da47f34ce924a12273. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"GPT-o3","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^I GRPO Attention Score 1: 27.82. Result ID: 69b902da47f34ce924a1228a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^I GRPO","World Model":"Qwen-7B^I"}. This record does not assert independent reproduction. ### Qwen-7B^I GRPO Attention Score 1: 26.67. Result ID: 69b902da47f34ce924a12289. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^I GRPO","World Model":"-"}. This record does not assert independent reproduction. ### GPT-o3 Attention Score 1: 25. Result ID: 69b902da47f34ce924a12274. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"GPT-o3","World Model":"GPT-o3"}. This record does not assert independent reproduction. ### MOCK Attention Score 1: 23.33. Result ID: 69b902da47f34ce924a12269. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"MOCK","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^I SFT Attention Score 1: 23.33. Result ID: 69b902da47f34ce924a12277. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^I SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^I SFT Attention Score 1: 23.21. Result ID: 69b902da47f34ce924a12278. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^I SFT","World Model":"Qwen-3B^I"}. This record does not assert independent reproduction. ### Qwen-7B^P&I SFT Attention Score 1: 21.41. Result ID: 69b902da47f34ce924a12286. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P&I SFT","World Model":"Qwen-7B^P&I"}. This record does not assert independent reproduction. ### Qwen-3B^P&I SFT Attention Score 1: 21.29. Result ID: 69b902da47f34ce924a1227a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P&I SFT","World Model":"Qwen-3B^P&I"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro Attention Score 1: 20. Result ID: 69b902da47f34ce924a12271. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Gemini-2.5-Pro","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P SFT Attention Score 1: 20. Result ID: 69b902da47f34ce924a12275. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P&I SFT Attention Score 1: 20. Result ID: 69b902da47f34ce924a12279. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P&I SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^P&I SFT Attention Score 1: 20. Result ID: 69b902da47f34ce924a12285. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P&I SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P SFT Attention Score 1: 19.87. Result ID: 69b902da47f34ce924a12276. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P SFT","World Model":"Qwen-3B^P"}. This record does not assert independent reproduction. ### Gemini-2.5-Pro Attention Score 1: 16.67. Result ID: 69b902da47f34ce924a12272. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Gemini-2.5-Pro","World Model":"Gemini-2.5-Pro"}. This record does not assert independent reproduction. ### Qwen-7B^P GRPO Attention Score 1: 16.67. Result ID: 69b902da47f34ce924a12287. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^P GRPO Attention Score 1: 15.68. Result ID: 69b902da47f34ce924a12288. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P GRPO","World Model":"Qwen-7B^P"}. This record does not assert independent reproduction. ### Qwen-7B^P SFT Attention Score 1: 13.57. Result ID: 69b902da47f34ce924a12282. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P SFT","World Model":"Qwen-7B^P"}. This record does not assert independent reproduction. ### Qwen-3B Attention Score 1: 13.33. Result ID: 69b902da47f34ce924a1226a. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-72B Attention Score 1: 13.33. Result ID: 69b902da47f34ce924a1226d. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-72B","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-3B^P GRPO Attention Score 1: 13.33. Result ID: 69b902da47f34ce924a1227b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^P SFT Attention Score 1: 13.33. Result ID: 69b902da47f34ce924a12281. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^P&I GRPO Attention Score 1: 13.33. Result ID: 69b902da47f34ce924a1228b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P&I GRPO","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^P&I GRPO Attention Score 1: 13.12. Result ID: 69b902da47f34ce924a1228c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^P&I GRPO","World Model":"Qwen-7B^P&I"}. This record does not assert independent reproduction. ### Qwen-3B^P GRPO Attention Score 1: 12.78. Result ID: 69b902da47f34ce924a1227c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-3B^P GRPO","World Model":"Qwen-3B^P"}. This record does not assert independent reproduction. ### Qwen-7B^I SFT Attention Score 1: 12.22. Result ID: 69b902da47f34ce924a12283. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^I SFT","World Model":"-"}. This record does not assert independent reproduction. ### Qwen-7B^I SFT Attention Score 1: 11.34. Result ID: 69b902da47f34ce924a12284. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-7B^I SFT","World Model":"Qwen-7B^I"}. This record does not assert independent reproduction. ### Qwen-72B Attention Score 1: 10. Result ID: 69b902da47f34ce924a1226e. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-72B","World Model":"Qwen-72B"}. This record does not assert independent reproduction. ### Qwen-32B Attention Score 1: 0.17. Result ID: 69b902da47f34ce924a1226c. [Reporting paper]() Recorded evaluation configuration: {"Policy Model":"Qwen-32B","World Model":"-"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.