Code Generation on LiveCodeBench (test)
53.6Pass@1 Overallo1-preview
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| o1-previewReasoning Strategy=Direct Reasoning2025.01 | 53.6 | — | — | — | — | — | |
| DeepSeek-R1-LiteReasoning Strategy=Direct Reasoning2025.01 | 51.6 | — | — | — | — | — | |
| CROWall (minutes)=1172026.05 | 51 | — | — | — | — | — | |
| GEPAWall (minutes)=732026.05 | 48.7 | — | — | — | — | — | |
| MetaHarnessWall (minutes)=2172026.05 | 40 | — | — | — | — | — | |
| Llama3.3-70BReasoning Strategy=Direct Reasoning2025.01 | 34.8 | 57.7 | 32.4 | 24.5 | — | — | |
| Focused-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 33.9 | 73.5 | 24.2 | 4.8 | — | — | |
| GPT-4oReasoning Strategy=Direct Reasoning2025.01 | 33.4 | — | — | — | — | — | |
| QwQ-32BReasoning Strategy=Direct Reasoning2025.01 | 33 | 61.5 | 29.7 | 20.4 | — | — | |
| Qwen2.5-72BReasoning Strategy=Direct Reasoning2025.01 | 33 | 53.8 | 29.7 | 24.5 | — | — | |
| Search-o1Reasoning Strategy=Reason-in-Documents2025.01 | 33 | 57.7 | 32.4 | 20.4 | — | — | |
| Token-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 32.5 | 70.6 | 23.9 | 3.7 | — | — | |
| Qwen2.5-coder-instruct-7BModel Type=Instruct2025.02 | 31.2 | 69.2 | 22 | 3.4 | — | — | |
| DPO / Step-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 31 | 68.5 | 23.3 | 1.9 | — | — | |
| Baseline2026.05 | 30.7 | — | — | — | — | — | |
| SFTBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 29.5 | 67 | 20.8 | 1.5 | — | — | |
| RAgent-QwQ-32BReasoning Strategy=RAG Agent2025.01 | 26.8 | 65.4 | 18.9 | 12.2 | — | — | |
| Focused-DPOBackbone=Qwen2.5-coder-base-7B, Model Type=Base2025.02 | 26.4 | 59.5 | 17.5 | 3 | — | — | |
| RAG-Qwen2.5-32BReasoning Strategy=Standard RAG2025.01 | 25.9 | 61.5 | 24.3 | 8.2 | — | — | |
| Token-DPOBackbone=Qwen2.5-coder-base-7B, Model Type=Base2025.02 | 25.3 | 58.4 | 16.3 | 2.2 | — | — | |
| SFTBackbone=Qwen2.5-coder-base-7B, Model Type=Base2025.02 | 25.1 | 58.4 | 15.7 | 2.2 | — | — | |
| Qwen2.5-Coder-32BReasoning Strategy=Direct Reasoning2025.01 | 25 | 61.5 | 16.2 | 12.2 | — | — | |
| DPO / Step-DPOBackbone=Qwen2.5-coder-base-7B, Model Type=Base2025.02 | 24.4 | 57.7 | 15.1 | 1.5 | — | — | |
| RAG-QwQ-32BReasoning Strategy=Standard RAG2025.01 | 24.1 | 57.7 | 16.2 | 12.2 | — | — | |
| RAgent-Qwen2.5-32BReasoning Strategy=RAG Agent2025.01 | 24.1 | 57.7 | 24.3 | 6.1 | — | — | |
| Qwen2.5-coder-base-7BModel Type=Base2025.02 | 24.1 | 56.7 | 15 | 1.7 | — | — | |
| Qwen2.5-32BReasoning Strategy=Direct Reasoning2025.01 | 22.3 | 42.3 | 18.9 | 14.3 | — | — | |
| Focused-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 21.3 | 51.3 | 11.8 | 1.9 | — | — | |
| Token-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 20.9 | 50.5 | 11.8 | 1.5 | — | — | |
| SFTBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 20.1 | 49.8 | 11.2 | 0.4 | — | — | |
| DPO / Step-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 19.8 | 49.1 | 10.9 | 0.4 | — | — | |
| Focused-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 19.7 | 47.7 | 10.6 | 1.9 | — | — | |
| MagiCoder-S-DS-6.7BModel Type=Instruct2025.02 | 19.3 | 48.1 | 10.7 | 0.1 | — | — | |
| Token-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 19.2 | 47 | 10 | 1.9 | — | — | |
| DPO / Step-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 18.4 | 46.2 | 9.4 | 0.7 | — | — | |
| SFTBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 18.3 | 46.2 | 9.4 | 0.4 | — | — | |
| DeepSeekCoder-instruct-6.7BModel Type=Instruct2025.02 | 18.1 | 45.3 | 9.1 | 0.9 | — | — | |
| Focused-DPOBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 16.9 | 42.3 | 8.5 | 1.1 | — | — | |
| SFTBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 16.6 | 41.9 | 8.2 | 0.7 | — | — | |
| Token-DPOBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 16.4 | 41.9 | 7.9 | 0.4 | — | — | |
| DPO / Step-DPOBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 16.1 | 41.2 | 7.9 | 0.4 | — | — | |
| DeepSeekCoder-base-6.7BModel Type=Base2025.02 | 15.5 | 39.9 | 7.4 | 0.4 | — | — | |
| Q-14B & A-14BModel Mode=Base, Search Strategy=Vanilla TTS2026.02 | — | — | — | — | 62.9 | 74.9 | |
| Q-14B & A-14BModel Mode=Base, Search Strategy=w/ RM2026.02 | — | — | — | — | 63.4 | 74.9 | |
| Q-14B & A-14BModel Mode=Base, Search Strategy=w/ all2026.02 | — | — | — | — | 65.1 | 78.9 | |
| Q-14B & A-14BModel Mode=Heter-MARS2, Search Strategy=Vanilla TTS2026.02 | — | — | — | — | 68.9 | 79.4 | |
| Q-14B & A-14BModel Mode=Heter-MARS2, Search Strategy=w/ RM2026.02 | — | — | — | — | 68.6 | 79.4 | |
| Q-14B & A-14BModel Mode=Heter-MARS2, Search Strategy=w/ all2026.02 | — | — | — | — | 70.3 | 80.6 | |
| Q-8B & A-8BModel Mode=Base, Search Strategy=Vanilla TTS2026.02 | — | — | — | — | 57.2 | 69.7 | |
| Q-8B & A-8BModel Mode=Base, Search Strategy=w/ RM2026.02 | — | — | — | — | 57.1 | 69.7 | |
| Q-8B & A-8BModel Mode=Base, Search Strategy=w/ all2026.02 | — | — | — | — | 58.9 | 71.4 | |
| Q-8B & A-8BModel Mode=Homo-MARS2, Search Strategy=Vanilla TTS2026.02 | — | — | — | — | 57.2 | 72.6 | |
| Q-8B & A-8BModel Mode=Homo-MARS2, Search Strategy=w/ RM2026.02 | — | — | — | — | 59.4 | 72.6 | |
| Q-8B & A-8BModel Mode=Homo-MARS2, Search Strategy=w/ all2026.02 | — | — | — | — | 60 | 69.1 | |
| Q-8B & A-8BModel Mode=Heter-MARS2, Search Strategy=Vanilla TTS2026.02 | — | — | — | — | 61.7 | 75.4 | |
| Q-8B & A-8BModel Mode=Heter-MARS2, Search Strategy=w/ RM2026.02 | — | — | — | — | 62.3 | 75.4 | |
| Q-8B & A-8BModel Mode=Heter-MARS2, Search Strategy=w/ all2026.02 | — | — | — | — | 62.9 | 72 |