Code Generation on HumanEval (Acc., TPF, TPS, Speedup vs. AR)
87.8AccuracyAR (greedy)
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| AR (greedy)Family=AR, Backbone=Qwen2.5-Coder-7B-Instruct, Hardware=single B200 GPU2025.12 | 87.8 | 100 | 83 | 1 | |
| JacobiFamily=AR + Jacobi, Backbone=Qwen2.5-Coder-7B-Instruct, Hardware=single B200 GPU2025.12 | 87.8 | 105 | 84.7 | 1.02 | |
| CLLMFamily=AR + Jacobi, Backbone=Qwen2.5-Coder-7B-Instruct, Hardware=single B200 GPU2025.12 | 87.8 | 268 | 207.4 | 2.5 | |
| Jacobi Forcing ModelFamily=AR + Jacobi, Backbone=Qwen2.5-Coder-7B-Instruct, Hardware=single B200 GPU2025.12 | 83.5 | 401 | 301.65 | 3.63 | |
| Jacobi Forcing Model (MR)Family=AR + Jacobi, Backbone=Qwen2.5-Coder-7B-Instruct, Hardware=single B200 GPU, Decoding Mode=multi-block decoding with rejection recycling2025.12 | 83.5 | 421 | 327.96 | 3.95 | |
| SDARFamily=dLLM, Hardware=single B200 GPU2025.12 | 78.7 | 236 | 31.46 | 0.38 | |
| EAGLE-3*Family=AR + Spec-Dec, Hardware=single B200 GPU2025.12 | 68.9 | 638 | 246.1 | 2.97 | |
| Fast-dLLM v2Family=dLLM, Hardware=single B200 GPU2025.12 | 63.4 | 100 | 83.29 | 1 | |
| HASS*Family=AR + Spec-Dec, Hardware=single B200 GPU2025.12 | 61.6 | 553 | 280.29 | 3.37 | |
| dParallelFamily=dLLM (distilled), Hardware=single B200 GPU2025.12 | 54.3 | 290 | 175.15 | 2.11 |