Code Generation on MBPP Plus (test)
83.6AccuracyTeam-of-Thoughts
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Team-of-Thoughtsorchestrator=GPT-5 Mini, context window=16,384 tokens2026.02 | 83.6 | — | |
| Claude Sonnet 4.5context window=4,096 tokens2026.02 | 83.33 | — | |
| GPT-5 Minicontext window=4,096 tokens2026.02 | 81.48 | — | |
| Qwen3-vl 235Bcontext window=4,096 tokens2026.02 | 80.69 | — | |
| AgentVersecontext window=4,096 tokens2026.02 | 79.37 | — | |
| DeepSeek v3.2context window=4,096 tokens2026.02 | 77.51 | — | |
| GPT-OSS 20Bcontext window=4,096 tokens2026.02 | 76.46 | — | |
| Focused-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 76.2 | 6.71 | |
| Token-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 75.1 | — | |
| DPO/Step-DPOBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 74.3 | — | |
| SFTBackbone=Qwen2.5-coder-instruct-7B, Model Type=Instruct2025.02 | 71.7 | — | |
| Qwen2.5-coder-instruct-7BModel Type=Instruct2025.02 | 71.4 | — | |
| GPT4-Turbo2023.12 | 70.7 | — | |
| Focused-DPOBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 70.4 | 3.03 | |
| Focused-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 69.8 | 4.76 | |
| GPT3.5-Turbo2023.12 | 69.4 | — | |
| Gemini 3 Flashcontext window=4,096 tokens2026.02 | 69.31 | — | |
| Token-DPOBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 69 | — | |
| DPO/Step-DPOBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 68.8 | — | |
| SFTBackbone=Qwen2.5-coder-base, Model Type=Base2025.02 | 68.8 | — | |
| Qwen2.5-coder-baseModel Type=Base2025.02 | 68.3 | — | |
| Token-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 68 | — | |
| DPO/Step-DPOBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 67.5 | — | |
| Focused-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 66.9 | 1.56 | |
| SFTBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 66.7 | — | |
| MagiCoder-S-DS-6.7BModel Type=Instruct2025.02 | 66.7 | — | |
| DPO/Step-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 66.1 | — | |
| Token-DPOBackbone=DeepSeekCoder-instruct-6.7B, Model Type=Instruct2025.02 | 66.1 | — | |
| DeepSeekCoder-instruct-6.7BModel Type=Instruct2025.02 | 65.9 | — | |
| SFTBackbone=MagiCoder-S-DS-6.7B, Model Type=Instruct2025.02 | 65.3 | — | |
| Phi-4context window=4,096 tokens2026.02 | 63.49 | — | |
| IF-CLPSize=34B, Open-source=weight&data, Base Model=CODELLAMA-PYTHON2023.12 | 62.7 | — | |
| IF-CLPSize=13B, Open-source=weight&data, Base Model=CODELLAMA-PYTHON2023.12 | 61.7 | — | |
| IF-CLSize=34B, Open-source=weight&data, Base Model=CODELLAMA2023.12 | 60.7 | — | |
| WizardCoder-CLPSize=34B, Open-source=weight2023.12 | 59.9 | — | |
| IF-CLSize=13B, Open-source=weight&data, Base Model=CODELLAMA2023.12 | 59.4 | — | |
| Focused-DPOBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 57.4 | 1.43 | |
| Token-DPOBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 57.4 | — | |
| DEARSetting=A, Student Model=Qwen2.5-1.5B-Instruct, Teacher Model=Qwen2.5-14B-Instruct2026.06 | 57.05 | — | |
| DPO/Step-DPOBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 56.9 | — | |
| MagicoderS-CLPSize=7B, Open-source=weight&data2023.12 | 56.6 | — | |
| DeepSeekCoder-base-6.7BModel Type=Base2025.02 | 56.6 | — | |
| SFTBackbone=DeepSeekCoder-base-6.7B, Model Type=Base2025.02 | 56.6 | — | |
| Qwen2.5-Coder 1.5B (ours)Model Size=1.5B, Training Stage=ours2026.05 | 55.6 | — | |
| Base modelSetting=A, Student Model=Qwen2.5-1.5B-Instruct, Teacher Model=Qwen2.5-14B-Instruct2026.06 | 53.36 | — | |
| CODELLAMA-PYTHONSize=34B, Open-source=weight2023.12 | 52.9 | — | |
| Standard OPDSetting=A, Student Model=Qwen2.5-1.5B-Instruct, Teacher Model=Qwen2.5-14B-Instruct2026.06 | 51.31 | — | |
| CODELLAMA-PYTHONSize=13B, Open-source=weight2023.12 | 50.9 | — | |
| Proxy (Tokenizer)CR=2.9, Model Size=14B2026.02 | 49.5 | — | |
| Proxy (Neural)CR=2.6, Model Size=14B2026.02 | 49.2 | — | |
| Tokenizer-basedCR=3.7, Model Size=14B2026.02 | 48.1 | — | |
| WizardCoder-CLPSize=7B, Open-source=weight2023.12 | 47.1 | — | |
| Tokenizer-basedCR=3.7, Model Size=4B2026.02 | 46.3 | — | |
| StarCoderSize=15B, Open-source=weight&data2023.12 | 46.1 | — | |
| Proxy (Tokenizer)CR=2.9, Model Size=7B2026.02 | 45.5 | — | |
| CODELLAMA-PYTHONSize=7B, Open-source=weight2023.12 | 45.4 | — | |
| Tokenizer-basedCR=3.7, Model Size=7B2026.02 | 45.2 | — | |
| CodeT5+Size=16B, Open-source=weight&data2023.12 | 44.4 | — | |
| Proxy (Tokenizer)CR=2.9, Model Size=4B2026.02 | 44.4 | — | |
| CodeGen-MonoSize=16B, Open-source=weight&data2023.12 | 43.6 | — | |
| Offline KDSetting=A, Student Model=Qwen2.5-1.5B-Instruct, Teacher Model=Qwen2.5-14B-Instruct2026.06 | 43.24 | — | |
| Byte-levelCR=1, Model Size=14B2026.02 | 42.1 | — | |
| Trans-LoRASource Model=Llama-2-7b, Target Model=Gemma-7b, Discriminator Model=Gemma-2b, Protocol=Zero-shot2024.05 | 42 | — | |
| Trans-LoRASource Model=Llama-2-7b, Target Model=Gemma-7b, Discriminator Model=Llama-2-7b, Protocol=Zero-shot2024.05 | 42 | — | |
| Byte-levelCR=1, Model Size=4B2026.02 | 41.8 | — | |
| Proxy (Neural)CR=2.6, Model Size=4B2026.02 | 41.8 | — | |
| Proxy (Neural)CR=2.6, Model Size=7B2026.02 | 41.8 | — | |
| Byte-levelCR=1, Model Size=7B2026.02 | 41.3 | — | |
| Qwen2.5-Coder 1.5B (base)Model Size=1.5B, Training Stage=base2026.05 | 41.3 | — | |
| Tokenizer-basedCR=3.7, Model Size=1.5B2026.02 | 41 | — | |
| MistralSize=7B, Open-source=weight2023.12 | 40.9 | — | |
| Trans-LoRASource Model=Gemma-2b, Target Model=Gemma-7b, Discriminator Model=Gemma-2b, Protocol=Zero-shot2024.05 | 40.6 | — | |
| Proxy (Tokenizer)CR=2.9, Model Size=1.5B2026.02 | 38.4 | — | |
| Tokenizer-based# Parameters=7B2026.02 | 36 | — | |
| Proxy (Neural)# Parameters=7B2026.02 | 36 | — | |
| Proxy (Tokenizer)# Parameters=7B2026.02 | 34.9 | — | |
| Trans-LoRASource Model=Llama-2-7b, Target Model=Llama-2-13b, Discriminator Model=Llama-2-7b, Protocol=Zero-shot2024.05 | 34.4 | — | |
| Source Model baselineModel=Gemma-2b, PEFT=LoRA, Protocol=Zero-shot2024.05 | 33.9 | — | |
| Byte-levelCR=1, Model Size=1.5B2026.02 | 33.6 | — | |
| Byte-level# Parameters=7B2026.02 | 32.5 | — | |
| Target Model baselineModel=Gemma-7b, PEFT=None, Protocol=Zero-shot2024.05 | 32.1 | — | |
| Target Model baselineModel=Llama-2-13b, PEFT=None, Protocol=Zero-shot2024.05 | 31.7 | — | |
| Proxy (Neural)CR=2.6, Model Size=1.5B2026.02 | 29.6 | — | |
| Tokenizer-basedCR=3.7, Model Size=0.5B2026.02 | 29.4 | — | |
| StarCoderSize=7B, Open-source=weight&data2023.12 | 28.8 | — | |
| Tokenizer-based# Parameters=1.5B2026.02 | 28 | — | |
| Byte-levelCR=1, Model Size=0.5B2026.02 | 25.9 | — | |
| Proxy (Tokenizer)CR=2.9, Model Size=0.5B2026.02 | 25.4 | — | |
| Proxy (Tokenizer)# Parameters=1.5B2026.02 | 25.1 | — | |
| Source Model baselineModel=Llama-2-7b, PEFT=LoRA, Protocol=Zero-shot2024.05 | 25 | — | |
| Proxy (Neural)# Parameters=1.5B2026.02 | 24.1 | — | |
| Byte-level# Parameters=1.5B2026.02 | 23.3 | — | |
| Proxy (Neural)CR=2.6, Model Size=0.5B2026.02 | 22 | — |