Summarization on CNNDM (Accept Rate, ROUGE-2)
12.64ROUGE-2Diversed
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| DiversedTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 12.64 | 58.46 | |
| DiversedTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 12.64 | 58.46 | |
| DiversedTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.04 | 12.37 | 81.92 | |
| DIVERSEDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 12.11 | 69.96 | |
| Static EnsembleTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 11.85 | 41.6 | |
| Static EnsembleTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 11.85 | 41.6 | |
| SpecCascadeTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 11.74 | 47.29 | |
| DIVERSEDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 11.65 | 0.9211 | |
| DiversedTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 11.6 | 64.26 | |
| AutoregressiveTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 11.48 | — | |
| AutoregressiveTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 11.48 | — | |
| DiversedTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.04 | 11.47 | 66.39 | |
| DiversedTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.04 | 11.47 | 66.39 | |
| Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 11.46 | 61.06 | |
| Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.04 | 11.45 | 85.04 | |
| SDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 11.38 | 36.4 | |
| SDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 11.38 | 36.4 | |
| DIVERSEDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 11.37 | 46.4 | |
| SpecCascadeTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 11.3 | 37.67 | |
| AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.04 | 11.29 | — | |
| SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.04 | 11.26 | 71.7 | |
| AutoregressiveTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 11.02 | — | |
| DIVERSEDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 10.97 | 46.59 | |
| Static EnsembleTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 10.9 | 40.67 | |
| Static EnsembleTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 10.88 | 41.07 | |
| AutoregressiveTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.04 | 10.87 | — | |
| AutoregressiveTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.04 | 10.87 | — | |
| DIVERSEDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 10.86 | 66.9 | |
| DIVERSEDTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 10.86 | 66.9 | |
| SpecCascadeTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 10.85 | 38.13 | |
| Static EnsembleTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.04 | 10.84 | 54.48 | |
| Static EnsembleTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.04 | 10.84 | 54.48 | |
| Static EnsembleTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 10.84 | 54.48 | |
| Static EnsembleTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 10.84 | 54.48 | |
| SD (lossy)Target Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 10.74 | 35.32 | |
| Static EnsembleTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 10.74 | 47.47 | |
| SpecCascadeTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 10.72 | 48.72 | |
| SD (Lossy)Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 10.71 | 48.73 | |
| SDTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.04 | 10.62 | 40.39 | |
| SDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.04 | 10.62 | 40.39 | |
| SD (Lossy)Target/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 10.62 | 45.43 | |
| SD (Lossy)Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 10.62 | 45.43 | |
| SD (lossy)Target/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 10.53 | 36.4 | |
| SD (Lossy)Target/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 10.51 | 38.86 | |
| Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 10.46 | 0.8758 | |
| SpecCascadeTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 10.42 | 51.44 | |
| SpecCascadeTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 10.42 | 51.44 | |
| SDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 10.35 | 34.19 | |
| SpecCascadeTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 10.19 | 0.522 | |
| SD (Lossy)Target/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 10.17 | 0.1785 | |
| AutoregressiveTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 9.97 | — | |
| AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 9.94 | — | |
| SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 9.92 | 0.1769 | |
| AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 9.86 | — | |
| SDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 9.85 | 35.26 | |
| SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 9.46 | 21.6 | |
| SDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 9.06 | 40.39 | |
| SDTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 9.06 | 40.39 | |
| AutoregressiveTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 9.01 | — | |
| AutoregressiveTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 9.01 | — | |
| AutoregressiveTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.04 | 9.01 | — | |
| SDTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04 | 9 | 44.27 |