Code Generation on MBPP (tokens/s, speedup)
32.93Tokens/sPACER
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PACERLLM Family/Size=Deepseek 1.3B/33B2026.02 | 32.93 | 1.88 | |
| SpeculativeLLM Family/Size=Deepseek 1.3B/33B2026.02 | 30.72 | 1.75 | |
| LookaheadLLM Family/Size=Deepseek 1.3B/33B2026.02 | 29.55 | 1.69 | |
| RESTLLM Family/Size=Deepseek 1.3B/33B2026.02 | 29.41 | 1.69 | |
| PACERLLM Family/Size=Deepseek 6.7B/33B2026.02 | 28.57 | 1.8 | |
| SpeculativeLLM Family/Size=Deepseek 6.7B/33B2026.02 | 26.62 | 1.68 | |
| AssistLLM Family/Size=Deepseek 1.3B/33B2026.02 | 26.6 | 1.52 | |
| RESTLLM Family/Size=Deepseek 6.7B/33B2026.02 | 25.64 | 1.6 | |
| LookaheadLLM Family/Size=Deepseek 6.7B/33B2026.02 | 23.67 | 1.49 | |
| AssistLLM Family/Size=Deepseek 6.7B/33B2026.02 | 21.07 | 1.33 | |
| PACERLLM Family/Size=Llama-2 7B/70B2026.02 | 19.9 | 2.14 | |
| SpeculativeLLM Family/Size=Llama-2 7B/70B2026.02 | 18.65 | 2.01 | |
| VanillaLLM Family/Size=Deepseek 1.3B/33B2026.02 | 17.53 | 1 | |
| AssistLLM Family/Size=Llama-2 7B/70B2026.02 | 16.3 | 1.76 | |
| VanillaLLM Family/Size=Deepseek 6.7B/33B2026.02 | 15.86 | 1 | |
| RESTLLM Family/Size=Llama-2 7B/70B2026.02 | 15.38 | 1.66 | |
| LookaheadLLM Family/Size=Llama-2 7B/70B2026.02 | 13.3 | 1.43 | |
| VanillaLLM Family/Size=Llama-2 7B/70B2026.02 | 9.28 | 1 |