# Hard Reasoning and Language Evaluation on HLE Source: [Hard Reasoning and Language Evaluation on HLE]() Collection index: [llms.txt]() Data updated: 2026-06-16T07:08:51.217Z Evaluation of model performance on the comprehensive Hard Reasoning and Language Evaluation task using the HLE dataset, reporting Accuracy, F-AE, Rounds, and Token metrics. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Hard Reasoning and Language Evaluation | | Dataset (including any stated split) | HLE | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 25 of 25 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. - [F-AE Score](): Higher is better; unit: SCALAR. - [Rounds](): Lower is better; unit: SCALAR. - [Tokens Used](): Lower is better; unit: SCALAR. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | Kimi-K2.6 | 54 | Not recorded | 6a30d866d33dfb23e783caa6 | | GLM-5.1 | 50.4 | Not recorded | 6a30d866d33dfb23e783caa6 | | Qwen-3.5 | 48.3 | Not recorded | 6a30d866d33dfb23e783caa6 | | DS-v4-Pro | 48.2 | Not recorded | 6a30d866d33dfb23e783caa6 | | DS-v4-Flash | 45.1 | Not recorded | 6a30d866d33dfb23e783caa6 | | DS-v4-Pro | 37.7 | Not recorded | 6a30d866d33dfb23e783caa6 | | Nemotron 3 Ultra | 37.4 | Not recorded | 6a30d866d33dfb23e783caa6 | | WebClipper (Hybrid) | 36.1 | Not recorded | 6992a46f9ebc499c97406ba0 | | Tongyi-DeepResearch | 35.8 | Not recorded | 6992a46f9ebc499c97406ba0 | | WebClipper (Eff) | 35.3 | Not recorded | 6992a46f9ebc499c97406ba0 | | Prompt Control | 34.9 | Not recorded | 6992a46f9ebc499c97406ba0 | | Kimi-K2.6 | 34.8 | Not recorded | 6a30d866d33dfb23e783caa6 | | Coarse Prune | 32.7 | Not recorded | 6992a46f9ebc499c97406ba0 | | DS-v4-Flash | 32.2 | Not recorded | 6a30d866d33dfb23e783caa6 | | Qwen-3.5 | 28.5 | Not recorded | 6a30d866d33dfb23e783caa6 | | GLM-5.1 | 27.2 | Not recorded | 6a30d866d33dfb23e783caa6 | | Nemotron 3 Ultra | 26.7 | Not recorded | 6a30d866d33dfb23e783caa6 | | OpenAI DeepResearch | 26.6 | Not recorded | 6992a46f9ebc499c97406ba0 | | OpenAI o3 | 24.9 | Not recorded | 6992a46f9ebc499c97406ba0 | | MiniMax-2.7 | 23.1 | Not recorded | 6a30d866d33dfb23e783caa6 | | Claude-4-Sonnet | 20.3 | Not recorded | 6992a46f9ebc499c97406ba0 | | Qwen3-235B-A22B-Instruct-2507 | 19.9 | Not recorded | 6992a46f9ebc499c97406ba0 | | Kimi-K2-Instruct-0905 | 14.6 | Not recorded | 6992a46f9ebc499c97406ba0 | | DeepSeek-R1-671B | 13.7 | Not recorded | 6992a46f9ebc499c97406ba0 | | WebExplorer | 11.6 | Not recorded | 6992a46f9ebc499c97406ba0 | ### Kimi-K2.6 Accuracy: 54. Result ID: 6a30f67efbc373592c2cae86. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Tools usage":"with tools","Parameter Count":"1T-A32B"}. This record does not assert independent reproduction. ### GLM-5.1 Accuracy: 50.4. Result ID: 6a30f67efbc373592c2cae84. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Tools usage":"with tools","Parameter Count":"744B-A40B"}. This record does not assert independent reproduction. ### Qwen-3.5 Accuracy: 48.3. Result ID: 6a30f67efbc373592c2cae88. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Tools usage":"with tools","Parameter Count":"397B-17B"}. This record does not assert independent reproduction. ### DS-v4-Pro Accuracy: 48.2. Result ID: 6a30f67efbc373592c2cae8a. [Reporting paper]() Recorded evaluation configuration: {"Tools usage":"with tools","Parameter Count":"1.6T-A49B"}. This record does not assert independent reproduction. ### DS-v4-Flash Accuracy: 45.1. Result ID: 6a30f67efbc373592c2cae8c. [Reporting paper]() Recorded evaluation configuration: {"Tools usage":"with tools","Parameter Count":"284B-A13B"}. This record does not assert independent reproduction. ### DS-v4-Pro Accuracy: 37.7. Result ID: 6a30f67efbc373592c2cae89. [Reporting paper]() Recorded evaluation configuration: {"Tools usage":"no tools","Parameter Count":"1.6T-A49B"}. This record does not assert independent reproduction. ### Nemotron 3 Ultra Accuracy: 37.4. Result ID: 6a30f67efbc373592c2cae81. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Tools usage":"with tools","Parameter Count":"550B-A55B"}. This record does not assert independent reproduction. ### WebClipper (Hybrid) Accuracy: 36.1. Result ID: 699a526e5698665ec4da0d36. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Pruning Method","Base Model":"Tongyi-DeepResearch","Variant":"Hybrid"}. This record does not assert independent reproduction. ### Tongyi-DeepResearch Accuracy: 35.8. Result ID: 699a526e5698665ec4da0d32. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Open-sourced Agent","Evaluation Environment":"Unified"}. This record does not assert independent reproduction. ### WebClipper (Eff) Accuracy: 35.3. Result ID: 699a526e5698665ec4da0d35. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Pruning Method","Base Model":"Tongyi-DeepResearch","Variant":"Efficiency"}. This record does not assert independent reproduction. ### Prompt Control Accuracy: 34.9. Result ID: 699a526e5698665ec4da0d33. [Reporting paper]() Recorded evaluation configuration: {"Category":"Pruning Method","Base Model":"Tongyi-DeepResearch"}. This record does not assert independent reproduction. ### Kimi-K2.6 Accuracy: 34.8. Result ID: 6a30f67efbc373592c2cae85. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Tools usage":"no tools","Parameter Count":"1T-A32B"}. This record does not assert independent reproduction. ### Coarse Prune Accuracy: 32.7. Result ID: 699a526e5698665ec4da0d34. [Reporting paper]() Recorded evaluation configuration: {"Category":"Pruning Method","Base Model":"Tongyi-DeepResearch"}. This record does not assert independent reproduction. ### DS-v4-Flash Accuracy: 32.2. Result ID: 6a30f67efbc373592c2cae8b. [Reporting paper]() Recorded evaluation configuration: {"Tools usage":"no tools","Parameter Count":"284B-A13B"}. This record does not assert independent reproduction. ### Qwen-3.5 Accuracy: 28.5. Result ID: 6a30f67efbc373592c2cae87. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Tools usage":"no tools","Parameter Count":"397B-17B"}. This record does not assert independent reproduction. ### GLM-5.1 Accuracy: 27.2. Result ID: 6a30f67efbc373592c2cae83. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Tools usage":"no tools","Parameter Count":"744B-A40B"}. This record does not assert independent reproduction. ### Nemotron 3 Ultra Accuracy: 26.7. Result ID: 6a30f67efbc373592c2cae80. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Tools usage":"no tools","Parameter Count":"550B-A55B"}. This record does not assert independent reproduction. ### OpenAI DeepResearch Accuracy: 26.6. Result ID: 699a526e5698665ec4da0d2c. [Reporting paper]() Recorded evaluation configuration: {"Category":"Close-sourced System"}. This record does not assert independent reproduction. ### OpenAI o3 Accuracy: 24.9. Result ID: 699a526e5698665ec4da0d2b. [Reporting paper]() Recorded evaluation configuration: {"Category":"Close-sourced System"}. This record does not assert independent reproduction. ### MiniMax-2.7 Accuracy: 23.1. Result ID: 6a30f67efbc373592c2cae82. [Reporting paper]() Recorded evaluation configuration: {"Tools usage":"no tools","Parameter Count":"230B-A10B"}. This record does not assert independent reproduction. ### Claude-4-Sonnet Accuracy: 20.3. Result ID: 699a526e5698665ec4da0d2d. [Reporting paper]() Recorded evaluation configuration: {"Category":"Close-sourced System"}. This record does not assert independent reproduction. ### Qwen3-235B-A22B-Instruct-2507 Accuracy: 19.9. Result ID: 699a526e5698665ec4da0d30. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Open-sourced Agent","Evaluation Environment":"Unified"}. This record does not assert independent reproduction. ### Kimi-K2-Instruct-0905 Accuracy: 14.6. Result ID: 699a526e5698665ec4da0d2e. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Open-sourced Agent","Evaluation Environment":"Unified"}. This record does not assert independent reproduction. ### DeepSeek-R1-671B Accuracy: 13.7. Result ID: 699a526e5698665ec4da0d2f. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Open-sourced Agent","Evaluation Environment":"Unified"}. This record does not assert independent reproduction. ### WebExplorer Accuracy: 11.6. Result ID: 699a526e5698665ec4da0d31. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Category":"Open-sourced Agent"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.