Code Generation on MBPP (pass@1, pass@5, NFE, Speed)
39Pass@1CAD
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CADGeneration=Block Diffusion (b=32), Model=LLaDA-8B-Instruct, Tokens/Step=1–32, shots=3, maximum sequence length (Lgen)=2562026.01 | 39 | 52.6 | 97.7 | 10.9 | 2.96 | |
| LLaDAGeneration=Block Diffusion (b=32), Model=LLaDA-8B-Instruct, Tokens/Step=1, shots=3, maximum sequence length (Lgen)=2562026.01 | 36.9 | 51.4 | 256 | 3.6 | 1 | |
| LLaDAGeneration=Sequential (b=1), Model=LLaDA-8B-Instruct, Tokens/Step=1, shots=3, maximum sequence length (Lgen)=2562026.01 | 36.8 | 49.2 | 256 | 3.6 | 1 | |
| CADGeneration=Pure Diffusion (b=256), Model=LLaDA-8B-Instruct, Tokens/Step=1–32, shots=3, maximum sequence length (Lgen)=2562026.01 | 36.4 | 51 | 99.4 | 10.8 | 2.96 | |
| LLaDAGeneration=Block Diffusion (b=32), Model=LLaDA-8B-Instruct, Tokens/Step=2, shots=3, maximum sequence length (Lgen)=2562026.01 | 33.2 | 44.8 | 128 | 7.3 | 2 | |
| LLaDAGeneration=Pure Diffusion (b=256), Model=LLaDA-8B-Instruct, Tokens/Step=2, shots=3, maximum sequence length (Lgen)=2562026.01 | 14.4 | 24.2 | 128 | 7.3 | 2 | |
| LLaDAGeneration=Pure Diffusion (b=256), Model=LLaDA-8B-Instruct, Tokens/Step=1, shots=3, maximum sequence length (Lgen)=2562026.01 | 6 | 14.8 | 256 | 3.7 | 1 |