Code Generation on MBPP (3-shot)
70.53Pass RateWeDLM-8B
Evaluation Results
| Method | Links | |
|---|---|---|
| WeDLM-8BInstruct Model=true, Model Category=WeDLM, Shot Configuration=3-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 70.53 | |
| Qwen3-8BInstruct Model=true, Model Category=AR Baseline, Shot Configuration=3-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=5122025.12 | 68.37 | |
| SDAR-8BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=3-shot, Inference Framework=JetEngine, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 67.97 | |
| Qwen2.5-7BInstruct Model=true, Model Category=AR Baseline, Shot Configuration=3-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=5122025.12 | 63.66 | |
| WeDLM-7BInstruct Model=true, Model Category=WeDLM, Shot Configuration=3-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 63.66 | |
| Dream-7BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=3-shot, Inference Framework=dInfer, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 58.52 | |
| SAS (Order Policy)Total generation length (L)=128, Block size (B)=162026.06 | 41.6 | |
| SAS (Order Policy)Total generation length (L)=256, Block size (B)=2562026.06 | 41 | |
| SAS (Order Policy)Total generation length (L)=256, Block size (B)=642026.06 | 40.7 | |
| SAS (Order Policy)Total generation length (L)=512, Block size (B)=5122026.06 | 40.7 | |
| SAS (Order Policy)Total generation length (L)=128, Block size (B)=322026.06 | 40.5 | |
| Confidence (Top-1)Total generation length (L)=128, Block size (B)=322026.06 | 40 | |
| SAS (Order Policy)Total generation length (L)=256, Block size (B)=322026.06 | 40 | |
| Confidence (Top-1)Total generation length (L)=256, Block size (B)=642026.06 | 39.6 | |
| EntropyTotal generation length (L)=128, Block size (B)=162026.06 | 39.6 | |
| Confidence (Top-1)Total generation length (L)=256, Block size (B)=2562026.06 | 39.5 | |
| SAS (Order Policy)Total generation length (L)=512, Block size (B)=1282026.06 | 39.5 | |
| SAS (Order Policy)Total generation length (L)=512, Block size (B)=642026.06 | 39.5 | |
| EntropyTotal generation length (L)=512, Block size (B)=642026.06 | 39.3 | |
| Confidence (Top-1)Total generation length (L)=512, Block size (B)=1282026.06 | 39.2 | |
| MarginTotal generation length (L)=128, Block size (B)=322026.06 | 39 | |
| EntropyTotal generation length (L)=512, Block size (B)=5122026.06 | 39 | |
| EntropyTotal generation length (L)=512, Block size (B)=1282026.06 | 39 | |
| Confidence (Top-1)Total generation length (L)=128, Block size (B)=162026.06 | 38.8 | |
| Confidence (Top-1)Total generation length (L)=512, Block size (B)=642026.06 | 38.8 | |
| MarginTotal generation length (L)=128, Block size (B)=162026.06 | 38.8 | |
| MarginTotal generation length (L)=512, Block size (B)=1282026.06 | 38.8 | |
| EntropyTotal generation length (L)=128, Block size (B)=322026.06 | 38.8 | |
| EntropyTotal generation length (L)=256, Block size (B)=2562026.06 | 38.8 | |
| Confidence (Top-1)Total generation length (L)=256, Block size (B)=322026.06 | 38.7 | |
| Confidence (Top-1)Total generation length (L)=512, Block size (B)=5122026.06 | 38.5 | |
| MarginTotal generation length (L)=512, Block size (B)=5122026.06 | 38.5 | |
| EntropyTotal generation length (L)=256, Block size (B)=642026.06 | 38.5 | |
| MarginTotal generation length (L)=256, Block size (B)=642026.06 | 38 | |
| MarginTotal generation length (L)=512, Block size (B)=642026.06 | 38 | |
| MarginTotal generation length (L)=256, Block size (B)=322026.06 | 37.8 | |
| EntropyTotal generation length (L)=256, Block size (B)=322026.06 | 37.8 | |
| SAS (Order Policy)Total generation length (L)=128, Block size (B)=1282026.06 | 37.5 | |
| Confidence (Top-1)Total generation length (L)=128, Block size (B)=1282026.06 | 37.3 | |
| MarginTotal generation length (L)=256, Block size (B)=2562026.06 | 36.3 | |
| LLaDA-8BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=3-shot, Inference Framework=dInfer, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.12 | 36.24 | |
| MarginTotal generation length (L)=128, Block size (B)=1282026.06 | 33.3 | |
| EntropyTotal generation length (L)=128, Block size (B)=1282026.06 | 32.3 |