Materials Science Scientific Reasoning on MatSciBench text-only
0.829Overall ScoreS1-NexusAgent
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| S1-NexusAgentFramework=Agent, Executor=Claude-4.5-Sonnet2026.02 | 0.829 | 0.817 | 0.819 | 0.832 | 0.828 | 0.824 | 0.843 | |
| S1-NexusAgentFramework=Agent, Executor=DeepSeek-V3.22026.02 | 0.796 | 0.818 | 0.765 | 0.796 | 0.783 | 0.788 | 0.809 | |
| Gemini-2.5-Pro (thinking)Framework=Base Model2026.02 | 0.7737 | 0.7849 | 0.7689 | 0.7715 | 0.7576 | 0.745 | 0.7869 | |
| S1-NexusAgentFramework=Agent, Executor=Qwen3-8B-Sci2026.02 | 0.756 | 0.766 | 0.725 | 0.756 | 0.714 | 0.752 | 0.763 | |
| S1-NexusAgentFramework=Agent, Executor=Qwen3-8B2026.02 | 0.726 | 0.728 | 0.691 | 0.721 | 0.712 | 0.738 | 0.735 | |
| Code-Aug.Framework=Agent, Backbone=Claude-3.7-Sonnet2026.02 | 0.7151 | 0.7208 | 0.6618 | 0.7075 | 0.6465 | 0.7089 | 0.7235 | |
| GPT-4.1Framework=Base Model2026.02 | 0.7073 | 0.6566 | 0.6891 | 0.7042 | 0.6162 | 0.6758 | 0.718 | |
| CoTFramework=Agent, Backbone=Claude-3.7-Sonnet2026.02 | 0.68 | 0.6679 | 0.6387 | 0.6711 | 0.6465 | 0.6542 | 0.6951 | |
| Claude-3.7-SonnetFramework=Base Model2026.02 | 0.6732 | 0.6566 | 0.6597 | 0.6589 | 0.6364 | 0.6484 | 0.6874 | |
| DeepSeek-V3Framework=Base Model2026.02 | 0.6615 | 0.6264 | 0.6197 | 0.6567 | 0.6364 | 0.6326 | 0.6689 | |
| CoTFramework=Agent, Backbone=DeepSeek-V32026.02 | 0.6439 | 0.6717 | 0.605 | 0.6291 | 0.6364 | 0.6326 | 0.6546 | |
| Code-Aug.Framework=Agent, Backbone=DeepSeek-V32026.02 | 0.6244 | 0.6151 | 0.5903 | 0.6269 | 0.6263 | 0.5793 | 0.6415 |