List your company
ResearchFormerly WizWand

Code Generation on MBPP (3-shot)

70.53Pass RateWeDLM-8B
30.770841.092951.41561.7371Dec 28, 2025Jan 26, 2026Feb 24, 2026Mar 26, 2026Apr 24, 2026May 23, 2026Jun 22, 2026
Updated 4mo ago

Evaluation results

MethodLinks
WeDLM-8BInstruct Model=true, Model Category=WeDLM, Shot Configuration=3-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.1270.53
Qwen3-8BInstruct Model=true, Model Category=AR Baseline, Shot Configuration=3-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=5122025.1268.37
SDAR-8BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=3-shot, Inference Framework=JetEngine, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.1267.97
Qwen2.5-7BInstruct Model=true, Model Category=AR Baseline, Shot Configuration=3-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=5122025.1263.66
WeDLM-7BInstruct Model=true, Model Category=WeDLM, Shot Configuration=3-shot, Inference Framework=vLLM, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.1263.66
Dream-7BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=3-shot, Inference Framework=dInfer, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.1258.52
SAS (Order Policy)Total generation length (L)=128, Block size (B)=162026.0641.6
SAS (Order Policy)Total generation length (L)=256, Block size (B)=2562026.0641
SAS (Order Policy)Total generation length (L)=256, Block size (B)=642026.0640.7
SAS (Order Policy)Total generation length (L)=512, Block size (B)=5122026.0640.7
SAS (Order Policy)Total generation length (L)=128, Block size (B)=322026.0640.5
Confidence (Top-1)Total generation length (L)=128, Block size (B)=322026.0640
SAS (Order Policy)Total generation length (L)=256, Block size (B)=322026.0640
Confidence (Top-1)Total generation length (L)=256, Block size (B)=642026.0639.6
EntropyTotal generation length (L)=128, Block size (B)=162026.0639.6
Confidence (Top-1)Total generation length (L)=256, Block size (B)=2562026.0639.5
SAS (Order Policy)Total generation length (L)=512, Block size (B)=1282026.0639.5
SAS (Order Policy)Total generation length (L)=512, Block size (B)=642026.0639.5
EntropyTotal generation length (L)=512, Block size (B)=642026.0639.3
Confidence (Top-1)Total generation length (L)=512, Block size (B)=1282026.0639.2
MarginTotal generation length (L)=128, Block size (B)=322026.0639
EntropyTotal generation length (L)=512, Block size (B)=5122026.0639
EntropyTotal generation length (L)=512, Block size (B)=1282026.0639
Confidence (Top-1)Total generation length (L)=128, Block size (B)=162026.0638.8
Confidence (Top-1)Total generation length (L)=512, Block size (B)=642026.0638.8
MarginTotal generation length (L)=128, Block size (B)=162026.0638.8
MarginTotal generation length (L)=512, Block size (B)=1282026.0638.8
EntropyTotal generation length (L)=128, Block size (B)=322026.0638.8
EntropyTotal generation length (L)=256, Block size (B)=2562026.0638.8
Confidence (Top-1)Total generation length (L)=256, Block size (B)=322026.0638.7
Confidence (Top-1)Total generation length (L)=512, Block size (B)=5122026.0638.5
MarginTotal generation length (L)=512, Block size (B)=5122026.0638.5
EntropyTotal generation length (L)=256, Block size (B)=642026.0638.5
MarginTotal generation length (L)=256, Block size (B)=642026.0638
MarginTotal generation length (L)=512, Block size (B)=642026.0638
MarginTotal generation length (L)=256, Block size (B)=322026.0637.8
EntropyTotal generation length (L)=256, Block size (B)=322026.0637.8
SAS (Order Policy)Total generation length (L)=128, Block size (B)=1282026.0637.5
Confidence (Top-1)Total generation length (L)=128, Block size (B)=1282026.0637.3
MarginTotal generation length (L)=256, Block size (B)=2562026.0636.3
LLaDA-8BInstruct Model=true, Model Category=DLLM Baseline, Shot Configuration=3-shot, Inference Framework=dInfer, Sampling Temperature=0.1, Max Generation Length=512, Window Size (W)=6, Penalty Coefficient (λ)=0.12025.1236.24
MarginTotal generation length (L)=128, Block size (B)=1282026.0633.3
EntropyTotal generation length (L)=128, Block size (B)=1282026.0632.3