Code Generation on MBPP (Throughput, Latency(s))
58Scored2Cache
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| d2CacheBackbone=Dream-Inst, n-shot=3-shot, Generation Length=5122025.09 | 58 | 12.47 | 40.32 | |
| IOAModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 52.56 | — | — | |
| dLLM-CacheBackbone=Dream-Inst, n-shot=3-shot, Generation Length=5122025.09 | 52.4 | 7.07 | 71.13 | |
| VanillaBackbone=Dream-Inst, n-shot=3-shot, Generation Length=5122025.09 | 52 | 2.73 | 182.78 | |
| Fast-dLLMBackbone=Dream-Inst, n-shot=3-shot, Generation Length=5122025.09 | 52 | 7.29 | 69.47 | |
| IOAModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 50.86 | — | — | |
| MADAModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 46.88 | — | — | |
| IOAModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 46.65 | — | — | |
| Star-AgentsModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 46.33 | — | — | |
| CasCoDModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 45.57 | — | — | |
| MADAModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 45.18 | — | — | |
| CounterDistillModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 45.05 | — | — | |
| DSSModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 44.73 | — | — | |
| Star-AgentsModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 44.63 | — | — | |
| CasCoDModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 43.87 | — | — | |
| LionModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 43.84 | — | — | |
| CounterDistillModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 43.35 | — | — | |
| DSSModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 43.03 | — | — | |
| LaMiniModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 42.19 | — | — | |
| LionModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 42.14 | — | — | |
| MADAModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 41.53 | — | — | |
| Star-AgentsModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 41.04 | — | — | |
| Self-InstructModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 40.97 | — | — | |
| LaMiniModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 40.49 | — | — | |
| CasCoDModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 40.42 | — | — | |
| CounterDistillModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 40.11 | — | — | |
| DSSModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 39.75 | — | — | |
| d2CacheBackbone=LLaDA-Inst, n-shot=3-shot, Generation Length=5122025.09 | 39.4 | 11.42 | 43.86 | |
| Self-InstructModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 39.27 | — | — | |
| LionModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 38.93 | — | — | |
| Fast-dLLMBackbone=LLaDA-Inst, n-shot=3-shot, Generation Length=5122025.09 | 38.4 | 6.8 | 73.27 | |
| VanillaBackbone=LLaDA-Inst, n-shot=3-shot, Generation Length=5122025.09 | 38 | 2.48 | 199.9 | |
| dLLM-CacheBackbone=LLaDA-Inst, n-shot=3-shot, Generation Length=5122025.09 | 38 | 6.97 | 71.79 | |
| LaMiniModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 37.34 | — | — | |
| UndistilledModel=Qwen2.5-14B, Teacher Model=OpenAI o12026.02 | 36.28 | — | — | |
| Self-InstructModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 36.22 | — | — | |
| UndistilledModel=Qwen2.5-7B, Teacher Model=OpenAI o12026.02 | 34.58 | — | — | |
| UndistilledModel=LLaMA3.1-8B, Teacher Model=OpenAI o12026.02 | 32.03 | — | — |