Code Generation on MBPP (Acc, TPF, AUP)
84.6Pass@1 AccuracyDS2
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| DS2Base model=Seed-Coder-8B, Eff. Tokens (%)=4.62026.06 | 84.6 | — | — | — | 66 | |
| CODEBLOCKBase model=Seed-Coder-8B, Eff. Tokens (%)=1.92026.06 | 83.9 | — | — | — | 65.5 | |
| CLAMBase model=Seed-Coder-8B, Eff. Tokens (%)=3.62026.06 | 83.3 | — | — | — | 64.9 | |
| RANDOM SELECTIONBase model=Seed-Coder-8B, Eff. Tokens (%)=1.92026.06 | 83.1 | — | — | — | 63.9 | |
| FULL TOKENSBase model=Seed-Coder-8B, Eff. Tokens (%)=100.02026.06 | 82.8 | — | — | — | 65.1 | |
| TOKEN CLEANINGBase model=Seed-Coder-8B, Eff. Tokens (%)=1.72026.06 | 82.3 | — | — | — | 62.8 | |
| BASEBase model=Seed-Coder-8B2026.06 | 82 | — | — | — | 62.5 | |
| RANDOM SELECTIONBase model=OpenCoder-8B-Base, Eff. Tokens (%)=1.92026.06 | 81.2 | — | — | — | 54.6 | |
| AR-OPD2026.06 | 80.4 | — | — | — | — | |
| CLAMBase model=OpenCoder-8B-Base, Eff. Tokens (%)=3.52026.06 | 80.1 | — | — | — | 54.6 | |
| DS2Base model=OpenCoder-8B-Base, Eff. Tokens (%)=4.62026.06 | 79.9 | — | — | — | 54.6 | |
| BASEBase model=OpenCoder-8B-Base2026.06 | 79.6 | — | — | — | 54.4 | |
| SFT2026.06 | 79.4 | — | — | — | — | |
| TOKEN CLEANINGBase model=OpenCoder-8B-Base, Eff. Tokens (%)=1.72026.06 | 79.4 | — | — | — | 51.3 | |
| CODEBLOCKBase model=OpenCoder-8B-Base, Eff. Tokens (%)=1.92026.06 | 78.8 | — | — | — | 57.1 | |
| Full OPD2026.06 | 78.5 | — | — | — | — | |
| FULL TOKENSBase model=OpenCoder-8B-Base, Eff. Tokens (%)=100.02026.06 | 78 | — | — | — | 54.5 | |
| Partial OPD2026.06 | 77.8 | — | — | — | — | |
| Base2026.06 | 77.5 | — | — | — | — | |
| CLAMBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=3.52026.06 | 76.7 | — | — | — | 51.8 | |
| DS2Base model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=4.62026.06 | 75.4 | — | — | — | 53.6 | |
| RANDOM SELECTIONBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=1.92026.06 | 74.6 | — | — | — | 53.6 | |
| CODEBLOCKBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=1.92026.06 | 74.1 | — | — | — | 54.6 | |
| FULL TOKENSBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=100.02026.06 | 73.3 | — | — | — | 52.7 | |
| BASEBase model=Qwen2.5-Coder-1.5B-Instruct2026.06 | 69.6 | — | — | — | 49.4 | |
| DASDBase Model=Qwen3-4B2026.05 | 63.2 | — | — | — | 74.3 | |
| TOKEN CLEANINGBase model=Qwen2.5-Coder-1.5B-Instruct, Eff. Tokens (%)=1.82026.06 | 62.4 | — | — | — | 49.1 | |
| OriginBase Model=Qwen3-4B2026.05 | 61.4 | — | — | — | 70.8 | |
| Hint-decodingBase Model=Qwen3-4B2026.05 | 57.4 | — | — | — | 71.6 | |
| Mask-PPLBase Model=Qwen3-4B2026.05 | 55.6 | — | — | — | 71.1 | |
| Self-DistillationBase Model=Qwen3-4B2026.05 | 53.2 | — | — | — | 69.7 | |
| DASDBase Model=Llama3.2-3B2026.05 | 50.4 | — | — | — | 56.7 | |
| Hint-decodingBase Model=Llama3.2-3B2026.05 | 50 | — | — | — | 56.3 | |
| Mask-PPLBase Model=Llama3.2-3B2026.05 | 49 | — | — | — | 54.7 | |
| OriginBase Model=Llama3.2-3B2026.05 | 47 | — | — | — | 52.3 | |
| Self-DistillationBase Model=Llama3.2-3B2026.05 | 46.6 | — | — | — | 53.1 | |
| NaRABackbone=LLaDA-8B-Instruct, Rank=322026.05 | 38.6 | — | — | — | — | |
| DASDBase Model=Gemma2-2B2026.05 | 38 | — | — | — | 47 | |
| LORABackbone=LLaDA-8B-Instruct, Rank=322026.05 | 37.2 | — | — | — | — | |
| PROMPT TUNINGBackbone=LLaDA-8B-Instruct, Rank=322026.05 | 37 | — | — | — | — | |
| OriginBase Model=Gemma2-2B2026.05 | 37 | — | — | — | 43 | |
| Hint-decodingBase Model=Gemma2-2B2026.05 | 37 | — | — | — | 45.9 | |
| ZERO-SHOTBackbone=LLaDA-8B-Instruct2026.05 | 36.4 | — | — | — | — | |
| Self-DistillationBase Model=Gemma2-2B2026.05 | 36.4 | — | — | — | 44.8 | |
| Mask-PPLBase Model=Gemma2-2B2026.05 | 36.4 | — | — | — | 45.4 | |
| P-TUNINGBackbone=LLaDA-8B-Instruct, Rank=322026.05 | 34.27 | — | — | — | — | |
| DeepCoder-1.5B-PreviewModel Type=Coder2026.06 | 30 | — | — | — | — | |
| DeepScaleR-1.5B-PreviewModel Type=Math2026.06 | 28.8 | — | — | — | — | |
| HIRABackbone=LLaDA-8B-Instruct, Rank=322026.05 | 27 | — | — | — | — | |
| SAR-MergingMerged Source=Math & Coder2026.06 | 22.4 | — | — | — | — | |
| RAM-MergingMerged Source=Math & Coder2026.06 | 12 | — | — | — | — | |
| DARE-MergingMerged Source=Math & Coder2026.06 | 11.6 | — | — | — | — | |
| TIES-MergingMerged Source=Math & Coder2026.06 | 10.8 | — | — | — | — | |
| Linear-MergingMerged Source=Math & Coder2026.06 | 10.6 | — | — | — | — | |
| Task ArithmeticMerged Source=Math & Coder2026.06 | 9.2 | — | — | — | — | |
| Base modelBackbone=Qwen2.5-0.5B-Instruct2026.05 | — | 17 | — | — | — | |
| D2FBackbone=LLaDA2026.05 | — | 38 | 1.94 | 53 | — | |
| D2F-LLaDAModel Category=dLLMs, Evaluation Protocol=3-shot2026.03 | — | 38 | 1.94 | 50 | — | |
| d3LLMBackbone=LLaDA2026.05 | — | 40.6 | 4.21 | 88.4 | — | |
| d3LLM-DreamModel Category=dLLMs, Evaluation Protocol=Zero-shot2026.03 | — | 55.6 | 2.96 | 141.4 | — | |
| d3LLM-LLaDAModel Category=dLLMs, Evaluation Protocol=3-shot2026.03 | — | 40.6 | 4.21 | 88.4 | — | |
| dParallelBackbone=LLaDA2026.05 | — | 40 | 2.35 | 60.5 | — | |
| dParallel-DreamModel Category=dLLMs, Evaluation Protocol=Zero-shot2026.03 | — | 55.4 | 2.24 | 108 | — | |
| dParallel-LLaDAModel Category=dLLMs, Evaluation Protocol=3-shot2026.03 | — | 40 | 2.35 | 60.5 | — | |
| DreamModel Category=dLLMs, Evaluation Protocol=Zero-shot2026.03 | — | 57.2 | 1 | 57.2 | — | |
| EAGLE-3 (LLaMA-3.1)Model Category=AR Models, Evaluation Protocol=Zero-shot2026.03 | — | 60.2 | 5.69 | 298.6 | — | |
| Fast-dLLMBackbone=LLaDA2026.05 | — | 38.6 | 2.13 | 56.6 | — | |
| Fast-dLLM-DreamModel Category=dLLMs, Evaluation Protocol=Zero-shot2026.03 | — | 53.2 | 1.2 | 63.6 | — | |
| Fast-dLLM-LLaDAModel Category=dLLMs, Evaluation Protocol=3-shot2026.03 | — | 38.6 | 2.13 | 56.6 | — | |
| Fast-dLLM-v2Model Category=Block-wise dLLMs, Evaluation Protocol=Zero-shot2026.03 | — | 50.1 | 2.04 | 81.9 | — | |
| LightningRL-8B-b32Model Category=Block-wise dLLMs, Evaluation Protocol=Zero-shot2026.03 | — | 58.3 | 11.1 | 641.6 | — | |
| LLaDAModel Category=dLLMs, Evaluation Protocol=3-shot2026.03 | — | 47.1 | 1 | 47.1 | — | |
| LLaDABackbone=LLaDA-Instruct2026.05 | — | 41.7 | 1 | 41.7 | — | |
| Qwen-2.5-7B-itModel Category=AR Models, Evaluation Protocol=Zero-shot2026.03 | — | 63.6 | 1 | 63.6 | — | |
| SDAR-8B-b32Model Category=Block-wise dLLMs, Evaluation Protocol=Zero-shot2026.03 | — | 58 | 2.44 | 81.1 | — | |
| SPD-QABackbone=Qwen2.5-0.5B-Instruct, Calibration Domain=QA2026.05 | — | 21 | — | — | — | |
| SSD-QABackbone=Qwen2.5-0.5B-Instruct, Calibration Domain=QA2026.05 | — | 12 | — | — | — | |
| TAD-QBackbone=LLaDA, Configuration=Quality2026.05 | — | 41.6 | 3.38 | 89.3 | — | |
| TAD-SBackbone=LLaDA, Configuration=Speed2026.05 | — | 40.6 | 3.42 | 80.3 | — |