Code Generation on HumanEval (Speed/Throughput)
287.77Tokens/sPRISM
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| PRISMModel=LLaMA-2, Temp=T = 02026.02 | 287.77 | — | — | 5.26 | — | |
| HASSModel=LLaMA-2, Temp=T = 02026.02 | 271.59 | — | — | 5.12 | — | |
| PRISMModel=LLaMA-3, Temp=T = 02026.02 | 252.18 | — | — | 5.44 | — | |
| EAGLE-2Model=LLaMA-2, Temp=T = 02026.02 | 251.94 | — | — | 4.75 | — | |
| PRISMModel=LLaMA-2, Temp=T = 12026.02 | 250.55 | — | — | 4.88 | — | |
| HASSModel=LLaMA-2, Temp=T = 12026.02 | 233.54 | — | — | 4.74 | — | |
| HASSModel=LLaMA-3, Temp=T = 02026.02 | 230.42 | — | — | 5.16 | — | |
| EAGLE-2Model=LLaMA-2, Temp=T = 12026.02 | 218.69 | — | — | 4.48 | — | |
| EAGLE-2Model=LLaMA-3, Temp=T = 02026.02 | 198.19 | — | — | 4.57 | — | |
| PRISMModel=LLaMA-3, Temp=T = 12026.02 | 182.53 | — | — | 5.08 | — | |
| StandardModel=LLaMA-3, Temp=T = 02026.02 | 171.76 | — | — | 5.93 | — | |
| HASSModel=LLaMA-3, Temp=T = 12026.02 | 169.27 | — | — | 4.75 | — | |
| EAGLE-2Model=LLaMA-3, Temp=T = 12026.02 | 154.91 | — | — | 4.35 | — | |
| StandardModel=LLaMA-3, Temp=T = 12026.02 | 154.6 | — | — | 5.36 | — | |
| StandardModel=LLaMA-2, Temp=T = 02026.02 | 133.32 | — | — | 2.72 | — | |
| StandardModel=LLaMA-2, Temp=T = 12026.02 | 129.76 | — | — | 2.62 | — | |
| VanillaModel=LLaMA-2, Temp=T = 02026.02 | 103.57 | — | — | — | — | |
| VanillaModel=LLaMA-2, Temp=T = 12026.02 | 103.27 | — | — | — | — | |
| Streaming-dLLMGen Length=256, Prompting Shot=0-shot2026.01 | 93.3 | 0.434 | 5.1 | — | — | |
| VanillaModel=LLaMA-3, Temp=T = 02026.02 | 91.51 | — | — | — | — | |
| SAM-Decoding[E2]Model=Llama3-8B2024.11 | 91.5 | 4.76 | 2.54 | — | — | |
| VanillaModel=LLaMA-3, Temp=T = 12026.02 | 91.02 | — | — | — | — | |
| EAGLE-2Model=Llama3-8B2024.11 | 85.58 | 4.74 | 2.37 | — | — | |
| SAM-Decoding[E2]Model=Vicuna-13B2024.11 | 78.96 | 5.12 | 3.54 | — | — | |
| SAM-Decoding[T]Model=Llama3-8B2024.11 | 78.04 | 2.77 | 2.16 | — | — | |
| EAGLE-2Model=Vicuna-13B2024.11 | 77.85 | 5.15 | 3.49 | — | — | |
| Streaming-dLLMGen Length=512, Prompting Shot=0-shot2026.01 | 75.8 | 0.445 | 6.4 | — | — | |
| Fast-dLLMGen Length=256, Prompting Shot=0-shot2026.01 | 71.5 | 0.433 | 3.9 | — | — | |
| Token RecyclingModel=Llama3-8B2024.11 | 71.49 | 2.93 | 1.99 | — | — | |
| SAM-DecodingModel=Llama3-8B2024.11 | 64.38 | 2.06 | 1.79 | — | — | |
| Fast-dLLMGen Length=512, Prompting Shot=0-shot2026.01 | 52 | 0.445 | 4.4 | — | — | |
| SAM-Decoding[T]Model=Vicuna-13B2024.11 | 50.87 | 2.79 | 2.28 | — | — | |
| SAGESetup=Llama3 8B-1B2026.01 | 50.2 | — | 2.67 | — | 4.89 | |
| SAM-DecodingModel=Vicuna-13B2024.11 | 48.92 | 2.42 | 2.2 | — | — | |
| Token RecyclingModel=Vicuna-13B2024.11 | 46.03 | 2.79 | 2.07 | — | — | |
| Native-SDSetup=Llama3 8B-1B2026.01 | 44.89 | — | 2.39 | — | 3.48 | |
| PLDModel=Llama3-8B2024.11 | 42.39 | 1.3 | 1.18 | — | — | |
| Streaming-dLLMGen Length=256, shots=0-shot2026.01 | 34.1 | 0.39 | 5.3 | — | — | |
| PLDModel=Vicuna-13B2024.11 | 32.06 | 1.54 | 1.44 | — | — | |
| Prefix-CacheGen Length=256, Prompting Shot=0-shot2026.01 | 30.6 | 0.427 | 1.7 | — | — | |
| SAM-Decoding[E2]Model=Vicuna-33B2024.11 | 29.56 | 3.61 | 3.14 | — | — | |
| EAGLE-2Model=Vicuna-33B2024.11 | 28.18 | 3.53 | 3 | — | — | |
| Streaming-dLLMGen Length=512, shots=0-shot2026.01 | 26.7 | 0.402 | 9.2 | — | — | |
| SAM-Decoding[T]Model=Vicuna-33B2024.11 | 22.44 | 2.73 | 2.39 | — | — | |
| Prefix-CacheGen Length=512, Prompting Shot=0-shot2026.01 | 21.8 | 0.457 | 1.8 | — | — | |
| Token RecyclingModel=Vicuna-33B2024.11 | 19.64 | 2.64 | 2.09 | — | — | |
| Fast-dLLMGen Length=256, shots=0-shot2026.01 | 19.1 | 0.372 | 3 | — | — | |
| SAM-DecodingModel=Vicuna-33B2024.11 | 19.08 | 2.05 | 2.03 | — | — | |
| dKV-CacheGen Length=256, Prompting Shot=0-shot2026.01 | 18.9 | 0.396 | 1 | — | — | |
| VanillaSetup=Llama3 8B-1B2026.01 | 18.8 | — | — | — | — | |
| LLaDAGen Length=256, Prompting Shot=0-shot2026.01 | 18.4 | 0.415 | 1 | — | — | |
| dKV-CacheGen Length=512, Prompting Shot=0-shot2026.01 | 14.3 | 0.451 | 1.2 | — | — | |
| PLDModel=Vicuna-33B2024.11 | 14.18 | 1.58 | 1.51 | — | — | |
| Fast-dLLMGen Length=512, shots=0-shot2026.01 | 13.6 | 0.398 | 4.7 | — | — | |
| LLaDAGen Length=512, Prompting Shot=0-shot2026.01 | 11.9 | 0.439 | 1 | — | — | |
| Prefix-CacheGen Length=256, shots=0-shot2026.01 | 10.9 | 0.384 | 1.7 | — | — | |
| dKV-CacheGen Length=256, shots=0-shot2026.01 | 6.6 | 0.402 | 1 | — | — | |
| LLaDA-1.5Gen Length=256, shots=0-shot2026.01 | 6.4 | 0.439 | 1 | — | — | |
| Prefix-CacheGen Length=512, shots=0-shot2026.01 | 4.8 | 0.378 | 1.7 | — | — | |
| dKV-CacheGen Length=512, shots=0-shot2026.01 | 3.3 | 0.402 | 1.1 | — | — | |
| LLaDA-1.5Gen Length=512, shots=0-shot2026.01 | 2.9 | 0.405 | 1 | — | — | |
| FullModel=LLaDA-8B, Inference Strategy=Full2026.03 | — | 30.5 | — | — | — | |
| FullModel=Dream-7B, Inference Strategy=Full2026.03 | — | 54.9 | — | — | — | |
| LSPModel=LLaDA-8B, Inference Strategy=LSP2026.03 | — | 29.3 | 1.22 | — | — | |
| LSPModel=Dream-7B, Inference Strategy=LSP2026.03 | — | 55.5 | 1.46 | — | — |