Code Generation on HumanEval (pass@1 and Avg.)
58.3pass@1Clean
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CleanModel=DeepSeek-Coder-6.7B, Quantization Precision=FP42026.06 | 58.3 | — | |
| PulseColModel=Dream, Attention Sparsity=50%2026.05 | 57.93 | 74.08 | |
| PulseColModel=Dream, Attention Sparsity=80%2026.05 | 57.32 | 72.83 | |
| Full AttentionModel=Dream2026.05 | 56.71 | 73.56 | |
| CleanModel=DeepSeek-Coder-6.7B, Quantization Precision=INT82026.06 | 56.4 | — | |
| StreamingLLMModel=Dream2026.05 | 55.49 | 53.23 | |
| QCBModel=DeepSeek-Coder-6.7B, Quantization Precision=INT82026.06 | 55 | — | |
| SparseDModel=Dream, Attention Sparsity=50%2026.05 | 54.87 | 70.85 | |
| QuantGuardModel=DeepSeek-Coder-6.7B, Quantization Precision=INT82026.06 | 54.2 | — | |
| QCBModel=DeepSeek-Coder-6.7B, Quantization Precision=FP42026.06 | 53.4 | — | |
| QCBModel=DeepSeek-Coder-6.7B, Quantization Precision=NF42026.06 | 53.2 | — | |
| QuantGuardModel=DeepSeek-Coder-6.7B, Quantization Precision=FP42026.06 | 51.6 | — | |
| QuantGuardModel=DeepSeek-Coder-6.7B, Quantization Precision=NF42026.06 | 51.6 | — | |
| CleanModel=Phi-2-2.7B, Quantization Precision=NF42026.06 | 51.2 | — | |
| CleanModel=DeepSeek-Coder-6.7B, Quantization Precision=NF42026.06 | 50.7 | — | |
| CleanModel=Phi-2-2.7B, Quantization Precision=INT82026.06 | 49.9 | — | |
| CleanModel=Phi-2-2.7B, Quantization Precision=FP42026.06 | 46.8 | — | |
| Sliding WindowModel=Dream2026.05 | 45.73 | 48.49 | |
| QuantGuardModel=Phi-2-2.7B, Quantization Precision=INT82026.06 | 45.2 | — | |
| QCBModel=Phi-2-2.7B, Quantization Precision=INT82026.06 | 44.1 | — | |
| QCBModel=Phi-2-2.7B, Quantization Precision=FP42026.06 | 43.3 | — | |
| QuantGuardModel=Phi-2-2.7B, Quantization Precision=FP42026.06 | 42.7 | — | |
| QuantGuardModel=Phi-2-2.7B, Quantization Precision=NF42026.06 | 41.4 | — | |
| QCBModel=Phi-2-2.7B, Quantization Precision=NF42026.06 | 40.6 | — | |
| Full AttentionModel=LLaDA-1.52026.05 | 39.63 | 67.58 | |
| SparseDModel=LLaDA-1.5, Attention Sparsity=50%2026.05 | 39.63 | 67.27 | |
| SparseDModel=LLaDA-1.5, Attention Sparsity=80%2026.05 | 39.02 | 63.89 | |
| PulseColModel=LLaDA-1.5, Attention Sparsity=50%2026.05 | 38.41 | 67.47 | |
| PulseColModel=LLaDA-1.5, Attention Sparsity=80%2026.05 | 38.41 | 66.58 | |
| StreamingLLMModel=LLaDA-1.52026.05 | 37.2 | 44.9 | |
| QuantGuardModel=Qwen2.5-Coder-1.5B, Quantization Precision=NF42026.06 | 35.4 | — | |
| QuantGuardModel=Qwen2.5-Coder-1.5B, Quantization Precision=INT82026.06 | 35.3 | — | |
| QCBModel=Qwen2.5-Coder-1.5B, Quantization Precision=NF42026.06 | 35 | — | |
| CleanModel=Qwen2.5-Coder-1.5B, Quantization Precision=NF42026.06 | 34.7 | — | |
| QCBModel=Qwen2.5-Coder-1.5B, Quantization Precision=INT82026.06 | 34.7 | — | |
| CleanModel=Qwen2.5-Coder-1.5B, Quantization Precision=INT82026.06 | 34.3 | — | |
| CleanModel=Qwen2.5-Coder-1.5B, Quantization Precision=FP42026.06 | 33.7 | — | |
| QuantGuardModel=Qwen2.5-Coder-1.5B, Quantization Precision=FP42026.06 | 32.4 | — | |
| QCBModel=Qwen2.5-Coder-1.5B, Quantization Precision=FP42026.06 | 31.1 | — | |
| Sliding WindowModel=LLaDA-1.52026.05 | 28.05 | 39.58 | |
| SparseDModel=Dream, Attention Sparsity=80%2026.05 | 17.68 | 49.85 | |
| QCBModel=StarCoder-1B, Quantization Precision=INT82026.06 | 17.3 | — | |
| QCBModel=StarCoder-1B, Quantization Precision=NF42026.06 | 16.5 | — | |
| QuantGuardModel=StarCoder-1B, Quantization Precision=INT82026.06 | 16.4 | — | |
| QuantGuardModel=StarCoder-1B, Quantization Precision=FP42026.06 | 16.3 | — | |
| QCBModel=StarCoder-1B, Quantization Precision=FP42026.06 | 15.9 | — | |
| QuantGuardModel=StarCoder-1B, Quantization Precision=NF42026.06 | 15.9 | — | |
| CleanModel=StarCoder-1B, Quantization Precision=INT82026.06 | 15 | — | |
| CleanModel=StarCoder-1B, Quantization Precision=NF42026.06 | 14.3 | — | |
| CleanModel=StarCoder-1B, Quantization Precision=FP42026.06 | 12.8 | — |