Code Generation on MBPP (Acc, NFE, T/F)
81.1AccuracyEvo 8B
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Evo 8BPost-training=SFT, Number of shots=42026.02 | 81.1 | — | — | |
| Qwen2.5 7BPost-training=SFT+RL, Number of shots=02026.02 | 79.2 | — | — | |
| Evo 8BShots=4, E2E Latency (s)=8.6, Inference Speed (tokens/s)=522026.02 | 77.4 | — | — | |
| Qwen2.5 7BShots=0, E2E Latency (s)=8.1, Inference Speed (tokens/s)=462026.02 | 75.2 | — | — | |
| MDLM 9BPost-training=SFT+RL, Number of shots=02026.02 | 74.9 | — | — | |
| BD3-LM 7BPost-training=SFT+RL, Number of shots=02026.02 | 67.2 | — | — | |
| Qwen3-8B (teacher)Role=Teacher2026.02 | 66 | — | — | |
| BD3-LM 7BShots=0, E2E Latency (s)=14.2, Inference Speed (tokens/s)=282026.02 | 63.7 | — | — | |
| LLaMA3 8BPost-training=SFT+RL, Number of shots=42026.02 | 57.6 | — | — | |
| MiniLLMTeacher=Qwen3-8B, Student=Qwen3-1.7B2026.02 | 52.1 | — | — | |
| TSD-KDTeacher=Qwen3-8B, Student=Qwen3-1.7B2026.02 | 51.9 | — | — | |
| GKDTeacher=Qwen3-8B, Student=Qwen3-1.7B2026.02 | 51.7 | — | — | |
| DistiLLMTeacher=Qwen3-8B, Student=Qwen3-1.7B2026.02 | 51.4 | — | — | |
| MDLM 7BShots=0, E2E Latency (s)=18.9, Inference Speed (tokens/s)=222026.02 | 51.3 | — | — | |
| Sequence KDTeacher=Qwen3-8B, Student=Qwen3-1.7B2026.02 | 51.3 | — | — | |
| Supervised KDTeacher=Qwen3-8B, Student=Qwen3-1.7B2026.02 | 51.2 | — | — | |
| Qwen3-1.7B (student)Role=Student2026.02 | 50.8 | — | — | |
| LLaMA3 8BShots=4, E2E Latency (s)=7.4, Inference Speed (tokens/s)=582026.02 | 47.1 | — | — | |
| LLaDA 8BShots=4, E2E Latency (s)=21.8, Inference Speed (tokens/s)=162026.02 | 38.5 | — | — | |
| Fast-dLLMGeneration length=256, Block size=322025.12 | 37.8 | 50.04 | 4.16 | |
| dUltra-codingGeneration length=256, Block size=32, Planner=Enabled2025.12 | 37.04 | 19.78 | 7.29 | |
| dParallelGeneration length=256, Block size=322025.12 | 36.2 | 22.77 | 6.3 | |
| LLaDA 8BPost-training=SFT, Number of shots=42026.02 | 34.2 | — | — | |
| d3LLMGeneration length=256, Block size=322025.12 | 33.2 | 32.26 | 6.9 | |
| ARD 7BPost-training=SFT+RL, Number of shots=42026.02 | 20.6 | — | — | |
| dUltra-coding (w/o planner)Generation length=256, Block size=32, Planner=Disabled2025.12 | 20.4 | 12.18 | 11.28 | |
| ARD 7BShots=4, E2E Latency (s)=32.5, Inference Speed (tokens/s)=122026.02 | 18.9 | — | — |