Code Generation on HumanEval+ v1 (test)
87.8Pass RateFocused-DPO
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Focused-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 87.8 | 4.41 | — | |
| Token-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 87.2 | — | — | |
| SFTBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 87.2 | — | — | |
| DPO/Step-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 85.4 | — | — | |
| Qwen2.5-coder-instruct-7BModel Type=Instruct2025.02 | 84.1 | — | — | |
| Focused-DPOBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 82.9 | 5.37 | — | |
| SFTBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 80.5 | — | — | |
| DPO/Step-DPOBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 79.9 | — | — | |
| Token-DPOBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 79.9 | — | — | |
| Qwen2.5-coder-baseModel Type=Base2025.02 | 78.7 | — | — | |
| Focused-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 74.4 | 8.93 | — | |
| Focused-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 73.2 | 4.38 | — | |
| Token-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 73.2 | — | — | |
| Token-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 72.6 | — | — | |
| SFTBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 72.6 | — | — | |
| DPO/Step-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 71.3 | — | — | |
| DeepSeekCoder-instruct-6.7BModel Type=Instruct2025.02 | 70.1 | — | — | |
| DPO/Step-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 70.1 | — | — | |
| SFTBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 70.1 | — | — | |
| MagiCoder-S-DS-6.7BModel Type=Instruct2025.02 | 68.3 | — | — | |
| Focused-DPOBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 42.7 | 7.79 | — | |
| Token-DPOBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 42.1 | — | — | |
| DeepSeekCoder-base-6.7BModel Type=Base2025.02 | 39.6 | — | — | |
| DPO/Step-DPOBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 39.6 | — | — | |
| SFTBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 39.6 | — | — | |
| Marginal SharpeningModel=Qwen3-4B, K=32, S=12026.05 | 0.927 | — | — | |
| Marginal SharpeningModel=Qwen3-4B, K=4, S=82026.05 | 0.915 | — | — | |
| OpenCodeInterpreter-DSSize=33B, Type=Instruct, Open-source Model=true, Open-source Data=true, Backbone=DeepSeekCoder, Synth. Human Feedback=Oracle2024.02 | 0.897 | — | 0.927 | |
| Power SamplingModel=Qwen3-4B2026.05 | 0.89 | — | — | |
| Majority VotingModel=Qwen3-4B, k=322026.05 | 0.884 | — | — | |
| Marginal SharpeningModel=Qwen3-8B, K=32, S=12026.05 | 0.878 | — | — | |
| OpenCodeInterpreter-DSSize=6.7B, Type=Instruct, Open-source Model=true, Open-source Data=true, Backbone=DeepSeekCoder, Synth. Human Feedback=true2024.02 | 0.866 | — | 0.872 | |
| Marginal SharpeningModel=Qwen3-8B, K=4, S=82026.05 | 0.86 | — | — | |
| Majority VotingModel=Qwen3-8B, k=322026.05 | 0.854 | — | — | |
| Temperature SamplingModel=Qwen3-4B2026.05 | 0.843 | — | — | |
| GPT-4 TurboOpen-source Model=false, Open-source Data=false, Execution Feedback=true2024.02 | 0.842 | — | 0.88 | |
| Temperature SamplingModel=Qwen3-8B2026.05 | 0.84 | — | — | |
| GPT-4 TurboOpen-source Model=false, Open-source Data=false2024.02 | 0.817 | — | 0.854 | |
| PrefillShareBackbone=Qwen3-8B-Base, KV Sharing=Supported2026.02 | 0.805 | — | — | |
| OpenCodeInterpreter-DSSize=6.7B, Type=Instruct, Open-source Model=true, Open-source Data=true, Backbone=DeepSeekCoder, Execution Feedback=true2024.02 | 0.787 | — | 0.811 | |
| Marginal SharpeningModel=Qwen3-1.7B, K=4, S=82026.05 | 0.762 | — | — | |
| Majority VotingModel=Qwen3-1.7B, k=322026.05 | 0.756 | — | — | |
| Marginal SharpeningModel=Qwen3-1.7B, K=32, S=12026.05 | 0.756 | — | — | |
| Full-FTBackbone=Qwen3-8B-Base, KV Sharing=Not Supported2026.02 | 0.743 | — | — | |
| Temperature SamplingModel=Qwen3-1.7B2026.05 | 0.722 | — | — | |
| OpenCodeInterpreter-DSSize=6.7B, Type=Instruct, Open-source Model=true, Open-source Data=true, Backbone=DeepSeekCoder2024.02 | 0.72 | — | 0.762 | |
| Power SamplingModel=Qwen3-1.7B2026.05 | 0.713 | — | — | |
| OpenCodeInterpreter-CLSize=7B, Type=Instruct, Open-source Model=true, Open-source Data=true, Backbone=CodeLlama, Execution Feedback=true2024.02 | 0.701 | — | 0.756 | |
| OpenCodeInterpreter-SC2Size=7B, Type=Instruct, Open-source Model=true, Open-source Data=true, Backbone=StarCoder2, Execution Feedback=true2024.02 | 0.695 | — | 0.756 | |
| OpenCodeInterpreter-SC2Size=7B, Type=Instruct, Open-source Model=true, Open-source Data=true, Backbone=StarCoder22024.02 | 0.689 | — | 0.738 | |
| OpenCodeInterpreter-CLSize=7B, Type=Instruct, Open-source Model=true, Open-source Data=true, Backbone=CodeLlama2024.02 | 0.677 | — | 0.726 | |
| Qwen3-8B-BaseKV Sharing=Inherent2026.02 | 0.616 | — | — | |
| Full-FTBackbone=LLaMA3.1-8B, KV Sharing=Not Supported2026.02 | 0.457 | — | — | |
| PrefillShareBackbone=LLaMA3.1-8B, KV Sharing=Supported2026.02 | 0.451 | — | — | |
| LLaMA3.1-8BKV Sharing=Inherent2026.02 | 0.299 | — | — |