Code Generation on MBPP (accuracy, throughput, speedup)
90.5AccuracyClaude 3.5 Sonnet
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Claude 3.5 SonnetEvaluation Protocol=3-shot CoT2026.03 | 90.5 | — | — | |
| Llama 3 405BEvaluation Protocol=3-shot CoT2026.03 | 88.6 | — | — | |
| GPT-4oEvaluation Protocol=3-shot CoT2026.03 | 87.8 | — | — | |
| Llama 3 70BEvaluation Protocol=3-shot CoT2026.03 | 86 | — | — | |
| Qwen3Parameters=8B2026.03 | 79 | — | — | |
| Qwen2.5Parameters=4B2026.03 | 78.8 | — | — | |
| LoopRPTParameters=2.6B2026.03 | 77.24 | — | — | |
| OuroParameters=2.6B2026.03 | 77.1 | — | — | |
| Qwen2.5Parameters=7B2026.03 | 73.8 | — | — | |
| Gemma3Parameters=12B2026.03 | 73.5 | — | — | |
| Llama 3 8BEvaluation Protocol=3-shot CoT2026.03 | 71.7 | — | — | |
| Base (r=1)Model scale=8B, r=12026.05 | 67.7 | 83 | — | |
| Spec (K=3)Model scale=8B, K=32026.05 | 67.3 | 1.42 | — | |
| CharacterFlywheel V7Evaluation Protocol=3-shot CoT2026.03 | 66.6 | — | — | |
| Base (r=1)Model scale=4B, r=12026.05 | 66.5 | 122.6 | — | |
| Spec (K=3)Model scale=4B, K=32026.05 | 66.5 | 1.31 | — | |
| Direct (K=1)Model scale=8B, K=12026.05 | 63.8 | 1.58 | — | |
| Gemma3Parameters=3B2026.03 | 63 | — | — | |
| Llama3.1Parameters=8B2026.03 | 62.4 | — | — | |
| Qwen3Parameters=4B2026.03 | 60.6 | — | — | |
| Latent Thinking OptimizationModel=Llama-3-8B2025.09 | 60 | — | — | |
| Direct (K=2)Model scale=8B, K=22026.05 | 59.9 | 1.8 | — | |
| DreamModel Variant=Instruct2026.01 | 58.8 | 4.7 | 1 | |
| Majority VotingModel=Llama-3-8B2025.09 | 57 | — | — | |
| DreamModel Variant=Base2026.01 | 56.2 | 4.7 | 1 | |
| Base ModelModel=Llama-3-8B2025.09 | 56 | — | — | |
| Direct (K=1)Model scale=4B, K=12026.05 | 55.6 | 1.62 | — | |
| Window-DiffusionModel Variant=Instruct, Window Size (L)=16, Refresh Cycle=32, Early Stopping=Disabled2026.01 | 55.4 | 31.1 | 6.6 | |
| Window-DiffusionModel Variant=Base, Window Size (L)=16, Refresh Cycle=32, Early Stopping=Disabled2026.01 | 54.4 | 29.7 | 6.3 | |
| Fast-dLLMModel Variant=Base, Caching Strategy=Prefix-Cache2026.01 | 53.8 | 15.9 | 3.4 | |
| Spec (K=3)Model scale=1.7B, K=32026.05 | 53.3 | 1.2 | — | |
| Fast-dLLMModel Variant=Base, Caching Strategy=Dual-Cache2026.01 | 53.2 | 25.3 | 5.4 | |
| DKV-CacheModel Variant=Instruct2026.01 | 53.2 | 6.4 | 1.4 | |
| Base (r=1)Model scale=1.7B, r=12026.05 | 52.9 | 213.1 | — | |
| DKV-CacheModel Variant=Base2026.01 | 51 | 10.1 | 2.2 | |
| Direct (K=2)Model scale=4B, K=22026.05 | 51 | 1.84 | — | |
| Llama3.2Parameters=3B2026.03 | 50.3 | — | — | |
| Fast-dLLMModel Variant=Instruct, Caching Strategy=Dual-Cache2026.01 | 50.2 | 24.9 | 5.3 | |
| Fast-dLLMModel Variant=Instruct, Caching Strategy=Prefix-Cache2026.01 | 48.6 | 10 | 2.1 | |
| Base (r=2)Model scale=8B, r=22026.05 | 48.2 | 1.69 | — | |
| CoDDDecoding Strategy=Entropy, Diffusion Steps=2562026.02 | 47.4 | — | — | |
| DreamDecoding Strategy=Entropy, Diffusion Steps=2562026.02 | 47.2 | — | — | |
| DreamDecoding Strategy=Margin, Diffusion Steps=2562026.02 | 45.6 | — | — | |
| Direct (K=1)Model scale=1.7B, K=12026.05 | 45.5 | 1.49 | — | |
| Direct (K=2)Model scale=1.7B, K=22026.05 | 42.4 | 1.63 | — | |
| DreamDecoding Strategy=Low Confidence, Diffusion Steps=2562026.02 | 42 | — | — | |
| wd1Gen Length=128, Steps=128, Block Size=322025.07 | 39.2 | — | — | |
| Base (r=2)Model scale=4B, r=22026.05 | 37.7 | 1.78 | — | |
| LLaDAGen Length=128, Steps=128, Block Size=322025.07 | 36.8 | — | — | |
| wd1Gen Length=512, Steps=512, Block Size=322025.07 | 36.8 | — | — | |
| wd1Gen Length=256, Steps=256, Block Size=322025.07 | 36.6 | — | — | |
| CoDDDecoding Strategy=Entropy, Diffusion Steps=1282026.02 | 36.6 | — | — | |
| LLaDAGen Length=512, Steps=512, Block Size=322025.07 | 36.4 | — | — | |
| DreamDecoding Strategy=Entropy, Diffusion Steps=1282026.02 | 36.2 | — | — | |
| LLaDAGen Length=256, Steps=256, Block Size=322025.07 | 35.6 | — | — | |
| CoDDDecoding Strategy=Margin, Diffusion Steps=2562026.02 | 35.6 | — | — | |
| DreamDecoding Strategy=Low Confidence, Diffusion Steps=1282026.02 | 35.6 | — | — | |
| LLaDADecoding Strategy=Margin, Diffusion Steps=2562026.02 | 35.2 | — | — | |
| LLaDADecoding Strategy=Low Confidence, Diffusion Steps=2562026.02 | 34.8 | — | — | |
| CoDDDecoding Strategy=Low Confidence, Diffusion Steps=2562026.02 | 34 | — | — | |
| Base (r=2)Model scale=1.7B, r=22026.05 | 32.3 | 1.64 | — | |
| Latent Thinking OptimizationModel=Llama-2-13B2025.09 | 32.2 | — | — | |
| DreamDecoding Strategy=Random, Diffusion Steps=1282026.02 | 32.2 | — | — | |
| DreamDecoding Strategy=Margin, Diffusion Steps=1282026.02 | 32.2 | — | — | |
| DreamDecoding Strategy=Random, Diffusion Steps=2562026.02 | 29.8 | — | — | |
| DreamDecoding Strategy=Random, Diffusion Steps=642026.02 | 29.4 | — | — | |
| CoDDDecoding Strategy=Entropy, Diffusion Steps=642026.02 | 27.8 | — | — | |
| DreamDecoding Strategy=Low Confidence, Diffusion Steps=642026.02 | 26.6 | — | — | |
| Majority VotingModel=Llama-2-13B2025.09 | 26.3 | — | — | |
| Base ModelModel=Llama-2-13B2025.09 | 24.7 | — | — | |
| DreamDecoding Strategy=Entropy, Diffusion Steps=642026.02 | 24.2 | — | — | |
| CoDDDecoding Strategy=Low Confidence, Diffusion Steps=1282026.02 | 23.8 | — | — | |
| Latent Thinking OptimizationModel=Llama-2-7B2025.09 | 23.7 | — | — | |
| DreamDecoding Strategy=Margin, Diffusion Steps=642026.02 | 23.6 | — | — | |
| CoDDDecoding Strategy=Margin, Diffusion Steps=1282026.02 | 22.8 | — | — | |
| LLaDADecoding Strategy=Margin, Diffusion Steps=1282026.02 | 21.6 | — | — | |
| CoDDDecoding Strategy=Random, Diffusion Steps=2562026.02 | 19.4 | — | — | |
| Majority VotingModel=Llama-2-7B2025.09 | 19.3 | — | — | |
| Base ModelModel=Llama-2-7B2025.09 | 18.9 | — | — | |
| LLaDADecoding Strategy=Random, Diffusion Steps=2562026.02 | 18 | — | — | |
| LLaDADecoding Strategy=Low Confidence, Diffusion Steps=1282026.02 | 17 | — | — | |
| CoDDDecoding Strategy=Random, Diffusion Steps=1282026.02 | 9.4 | — | — | |
| CoDDDecoding Strategy=Margin, Diffusion Steps=642026.02 | 8.8 | — | — | |
| LLaDADecoding Strategy=Margin, Diffusion Steps=642026.02 | 7.8 | — | — | |
| CoDDDecoding Strategy=Low Confidence, Diffusion Steps=642026.02 | 7.2 | — | — | |
| LLaDADecoding Strategy=Random, Diffusion Steps=1282026.02 | 6.2 | — | — | |
| LLaDADecoding Strategy=Low Confidence, Diffusion Steps=642026.02 | 5.4 | — | — | |
| CoDDDecoding Strategy=Random, Diffusion Steps=642026.02 | 4 | — | — | |
| LLaDADecoding Strategy=Random, Diffusion Steps=642026.02 | 2.2 | — | — |