Summarization on XSum (Accept Rate, ROUGE-2)
9.16ROUGE-2Autoregressive
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| AutoregressiveTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.04 | 9.16 | — | |
| AutoregressiveTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.04 | 9.16 | — | |
| SDTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.04 | 9.04 | 35.54 | |
| SDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.04 | 9.04 | 35.54 | |
| AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.04 | 8.42 | — | |
| SD (Lossy)Target/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 8.38 | 39.48 | |
| SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.04 | 8.36 | 50.24 | |
| AutoregressiveTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 8.31 | — | |
| SDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 8.27 | 35.76 | |
| SpecCascadeTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 8.27 | 39.42 | |
| SD (Lossy)Target/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 7.97 | 38.83 | |
| SD (Lossy)Target/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 7.96 | 40.91 | |
| Static EnsembleTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.04 | 7.9 | 56.31 | |
| Static EnsembleTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.04 | 7.9 | 56.31 | |
| Static EnsembleTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 7.9 | 61.42 | |
| SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 7.82 | 37.67 | |
| DiversedTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.04 | 7.66 | 94.81 | |
| SpecCascadeTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 7.61 | 43.95 | |
| AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 7.56 | — | |
| SpecCascadeTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 7.53 | 40.22 | |
| Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.04 | 7.53 | 61.44 | |
| DIVERSEDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 7.52 | 64.75 | |
| Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.04 | 7.31 | 51.2 | |
| DIVERSEDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 7.23 | 70.53 | |
| DiversedTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.04 | 7.22 | 63.38 | |
| DiversedTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.04 | 7.22 | 63.38 | |
| Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 7.22 | 51.58 | |
| DIVERSEDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.04 | 7.22 | 63.38 | |
| SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 7.09 | 20.44 | |
| AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.04 | 7.03 | — | |
| DiversedTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 5.54 | 38.12 | |
| Static EnsembleTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 5.39 | 34.14 | |
| AutoregressiveTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 5.38 | — | |
| SDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.04 | 5.35 | 19.56 | |
| SDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 5.26 | 16.45 | |
| DIVERSEDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 5.25 | 45.87 | |
| AutoregressiveTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 5.21 | — | |
| Static EnsembleTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 5.19 | 30.8 | |
| DIVERSEDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 5.01 | 47.01 | |
| Static EnsembleTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 4.98 | 21.24 | |
| SpecCascadeTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 4.97 | 19.45 | |
| SDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 4.95 | 15.6 | |
| SpecCascadeTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 4.91 | 32.98 | |
| SD (lossy)Target Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.04 | 4.9 | 19.56 | |
| AutoregressiveTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 4.9 | — | |
| SD (lossy)Target/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.04 | 4.9 | 19.56 |