List your company
ResearchFormerly WizWand

Code Generation on MBPP (accuracy, throughput, speedup)

90.5AccuracyClaude 3.5 Sonnet
-1.33222.50946.3570.191Jul 7, 2025Aug 27, 2025Oct 18, 2025Dec 8, 2025Jan 29, 2026Mar 21, 2026May 12, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Claude 3.5 SonnetEvaluation Protocol=3-shot CoT2026.0390.5——
Llama 3 405BEvaluation Protocol=3-shot CoT2026.0388.6——
GPT-4oEvaluation Protocol=3-shot CoT2026.0387.8——
Llama 3 70BEvaluation Protocol=3-shot CoT2026.0386——
Qwen3Parameters=8B2026.0379——
Qwen2.5Parameters=4B2026.0378.8——
LoopRPTParameters=2.6B2026.0377.24——
OuroParameters=2.6B2026.0377.1——
Qwen2.5Parameters=7B2026.0373.8——
Gemma3Parameters=12B2026.0373.5——
Llama 3 8BEvaluation Protocol=3-shot CoT2026.0371.7——
Base (r=1)Model scale=8B, r=12026.0567.783—
Spec (K=3)Model scale=8B, K=32026.0567.31.42—
CharacterFlywheel V7Evaluation Protocol=3-shot CoT2026.0366.6——
Base (r=1)Model scale=4B, r=12026.0566.5122.6—
Spec (K=3)Model scale=4B, K=32026.0566.51.31—
Direct (K=1)Model scale=8B, K=12026.0563.81.58—
Gemma3Parameters=3B2026.0363——
Llama3.1Parameters=8B2026.0362.4——
Qwen3Parameters=4B2026.0360.6——
Latent Thinking OptimizationModel=Llama-3-8B2025.0960——
Direct (K=2)Model scale=8B, K=22026.0559.91.8—
DreamModel Variant=Instruct2026.0158.84.71
Majority VotingModel=Llama-3-8B2025.0957——
DreamModel Variant=Base2026.0156.24.71
Base ModelModel=Llama-3-8B2025.0956——
Direct (K=1)Model scale=4B, K=12026.0555.61.62—
Window-DiffusionModel Variant=Instruct, Window Size (L)=16, Refresh Cycle=32, Early Stopping=Disabled2026.0155.431.16.6
Window-DiffusionModel Variant=Base, Window Size (L)=16, Refresh Cycle=32, Early Stopping=Disabled2026.0154.429.76.3
Fast-dLLMModel Variant=Base, Caching Strategy=Prefix-Cache2026.0153.815.93.4
Spec (K=3)Model scale=1.7B, K=32026.0553.31.2—
Fast-dLLMModel Variant=Base, Caching Strategy=Dual-Cache2026.0153.225.35.4
DKV-CacheModel Variant=Instruct2026.0153.26.41.4
Base (r=1)Model scale=1.7B, r=12026.0552.9213.1—
DKV-CacheModel Variant=Base2026.015110.12.2
Direct (K=2)Model scale=4B, K=22026.05511.84—
Llama3.2Parameters=3B2026.0350.3——
Fast-dLLMModel Variant=Instruct, Caching Strategy=Dual-Cache2026.0150.224.95.3
Fast-dLLMModel Variant=Instruct, Caching Strategy=Prefix-Cache2026.0148.6102.1
Base (r=2)Model scale=8B, r=22026.0548.21.69—
CoDDDecoding Strategy=Entropy, Diffusion Steps=2562026.0247.4——
DreamDecoding Strategy=Entropy, Diffusion Steps=2562026.0247.2——
DreamDecoding Strategy=Margin, Diffusion Steps=2562026.0245.6——
Direct (K=1)Model scale=1.7B, K=12026.0545.51.49—
Direct (K=2)Model scale=1.7B, K=22026.0542.41.63—
DreamDecoding Strategy=Low Confidence, Diffusion Steps=2562026.0242——
wd1Gen Length=128, Steps=128, Block Size=322025.0739.2——
Base (r=2)Model scale=4B, r=22026.0537.71.78—
LLaDAGen Length=128, Steps=128, Block Size=322025.0736.8——
wd1Gen Length=512, Steps=512, Block Size=322025.0736.8——
wd1Gen Length=256, Steps=256, Block Size=322025.0736.6——
CoDDDecoding Strategy=Entropy, Diffusion Steps=1282026.0236.6——
LLaDAGen Length=512, Steps=512, Block Size=322025.0736.4——
DreamDecoding Strategy=Entropy, Diffusion Steps=1282026.0236.2——
LLaDAGen Length=256, Steps=256, Block Size=322025.0735.6——
CoDDDecoding Strategy=Margin, Diffusion Steps=2562026.0235.6——
DreamDecoding Strategy=Low Confidence, Diffusion Steps=1282026.0235.6——
LLaDADecoding Strategy=Margin, Diffusion Steps=2562026.0235.2——
LLaDADecoding Strategy=Low Confidence, Diffusion Steps=2562026.0234.8——
CoDDDecoding Strategy=Low Confidence, Diffusion Steps=2562026.0234——
Base (r=2)Model scale=1.7B, r=22026.0532.31.64—
Latent Thinking OptimizationModel=Llama-2-13B2025.0932.2——
DreamDecoding Strategy=Random, Diffusion Steps=1282026.0232.2——
DreamDecoding Strategy=Margin, Diffusion Steps=1282026.0232.2——
DreamDecoding Strategy=Random, Diffusion Steps=2562026.0229.8——
DreamDecoding Strategy=Random, Diffusion Steps=642026.0229.4——
CoDDDecoding Strategy=Entropy, Diffusion Steps=642026.0227.8——
DreamDecoding Strategy=Low Confidence, Diffusion Steps=642026.0226.6——
Majority VotingModel=Llama-2-13B2025.0926.3——
Base ModelModel=Llama-2-13B2025.0924.7——
DreamDecoding Strategy=Entropy, Diffusion Steps=642026.0224.2——
CoDDDecoding Strategy=Low Confidence, Diffusion Steps=1282026.0223.8——
Latent Thinking OptimizationModel=Llama-2-7B2025.0923.7——
DreamDecoding Strategy=Margin, Diffusion Steps=642026.0223.6——
CoDDDecoding Strategy=Margin, Diffusion Steps=1282026.0222.8——
LLaDADecoding Strategy=Margin, Diffusion Steps=1282026.0221.6——
CoDDDecoding Strategy=Random, Diffusion Steps=2562026.0219.4——
Majority VotingModel=Llama-2-7B2025.0919.3——
Base ModelModel=Llama-2-7B2025.0918.9——
LLaDADecoding Strategy=Random, Diffusion Steps=2562026.0218——
LLaDADecoding Strategy=Low Confidence, Diffusion Steps=1282026.0217——
CoDDDecoding Strategy=Random, Diffusion Steps=1282026.029.4——
CoDDDecoding Strategy=Margin, Diffusion Steps=642026.028.8——
LLaDADecoding Strategy=Margin, Diffusion Steps=642026.027.8——
CoDDDecoding Strategy=Low Confidence, Diffusion Steps=642026.027.2——
LLaDADecoding Strategy=Random, Diffusion Steps=1282026.026.2——
LLaDADecoding Strategy=Low Confidence, Diffusion Steps=642026.025.4——
CoDDDecoding Strategy=Random, Diffusion Steps=642026.024——
LLaDADecoding Strategy=Random, Diffusion Steps=642026.022.2——