Scientific Code Generation on SciCode (test)
43.8Subset AccuracyLCP
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LCPModel=Claude-sonnet-4, Knowledge Condition=With Knowledge2026.03 | 43.8 | 13.8 | |
| LCPModel=Deepseek-R1, Knowledge Condition=With Knowledge2026.03 | 43.1 | 13.8 | |
| LCPModel=Claude-sonnet-4, Knowledge Condition=Without Knowledge2026.03 | 42.7 | 13.8 | |
| LCPModel=Deepseek-v3, Knowledge Condition=With Knowledge2026.03 | 42.4 | 12.3 | |
| LCPModel=Qwen3-235B-A22b-Instruct, Knowledge Condition=With Knowledge2026.03 | 41 | 10.8 | |
| LCPModel=Deepseek-R1, Knowledge Condition=Without Knowledge2026.03 | 41 | 10.8 | |
| LCPModel=GPT-4o, Knowledge Condition=With Knowledge2026.03 | 40.6 | 10.8 | |
| LCPModel=Deepseek-v3, Knowledge Condition=Without Knowledge2026.03 | 40.3 | 10.8 | |
| LCPModel=Qwen3-235B-A22b-Instruct, Knowledge Condition=Without Knowledge2026.03 | 38.9 | 9.2 | |
| BaselineModel=Deepseek-v3, Knowledge Condition=With Knowledge2026.03 | 38.8 | 10.8 | |
| BaselineModel=Claude-sonnet-4, Knowledge Condition=With Knowledge2026.03 | 38.8 | 10.8 | |
| LCPModel=Qwen3-next-80b a3b-instruct, Knowledge Condition=With Knowledge2026.03 | 38.5 | 10.8 | |
| BaselineModel=Deepseek-R1, Knowledge Condition=With Knowledge2026.03 | 37.8 | 10.8 | |
| LCPModel=Qwen3-next-80b a3b-instruct, Knowledge Condition=Without Knowledge2026.03 | 37.5 | 9.2 | |
| BaselineModel=Qwen3-235B-A22b-Instruct, Knowledge Condition=With Knowledge2026.03 | 37.2 | 10.8 | |
| LCPModel=GPT-4o, Knowledge Condition=Without Knowledge2026.03 | 37.2 | 7.7 | |
| LCPModel=Qwen3-32b, Knowledge Condition=With Knowledge2026.03 | 36.1 | 7.7 | |
| BaselineModel=GPT-4o, Knowledge Condition=With Knowledge2026.03 | 33.7 | 7.7 | |
| LCPModel=Qwen3-32b, Knowledge Condition=Without Knowledge2026.03 | 33 | 6.2 | |
| LCPModel=Qwen3-14b, Knowledge Condition=With Knowledge2026.03 | 32.6 | 6.2 | |
| BaselineModel=Qwen3-next-80b a3b-instruct, Knowledge Condition=With Knowledge2026.03 | 32.6 | 12.3 | |
| BaselineModel=Claude-sonnet-4, Knowledge Condition=Without Knowledge2026.03 | 31.3 | 7.7 | |
| LCPModel=Qwen3-14b, Knowledge Condition=Without Knowledge2026.03 | 30.6 | 6.2 | |
| BaselineModel=Qwen3-235B-A22b-Instruct, Knowledge Condition=Without Knowledge2026.03 | 30.6 | 4.6 | |
| BaselineModel=Deepseek-R1, Knowledge Condition=Without Knowledge2026.03 | 29.6 | 4.6 | |
| BaselineModel=Deepseek-v3, Knowledge Condition=Without Knowledge2026.03 | 27.8 | 3.1 | |
| LCPModel=Qwen3-8b, Knowledge Condition=With Knowledge2026.03 | 27.4 | 4.6 | |
| BaselineModel=Qwen3-32b, Knowledge Condition=With Knowledge2026.03 | 27.4 | 7.7 | |
| BaselineModel=Qwen3-14b, Knowledge Condition=With Knowledge2026.03 | 25 | 6.2 | |
| LCPModel=Qwen3-8b, Knowledge Condition=Without Knowledge2026.03 | 24.7 | 4.6 | |
| BaselineModel=GPT-4o, Knowledge Condition=Without Knowledge2026.03 | 24.1 | 1.5 | |
| BaselineModel=Qwen3-next-80b a3b-instruct, Knowledge Condition=Without Knowledge2026.03 | 21.5 | 3.1 | |
| BaselineModel=Qwen3-8b, Knowledge Condition=With Knowledge2026.03 | 19.8 | 1.5 | |
| BaselineModel=Qwen3-32b, Knowledge Condition=Without Knowledge2026.03 | 18.4 | 0 | |
| BaselineModel=Qwen3-14b, Knowledge Condition=Without Knowledge2026.03 | 17.7 | 1.5 | |
| BaselineModel=Qwen3-8b, Knowledge Condition=Without Knowledge2026.03 | 13.2 | 0 |