List your company
ResearchFormerly WizWand

Summarization on XSum (Accept Rate, ROUGE-2)

9.16ROUGE-2Autoregressive
4.72965.87987.038.1802Apr 8, 2026
Updated 6mo ago

Evaluation results

MethodLinks
AutoregressiveTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.049.16—
AutoregressiveTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.049.16—
SDTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.049.0435.54
SDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.049.0435.54
AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.048.42—
SD (Lossy)Target/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.048.3839.48
SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.048.3650.24
AutoregressiveTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.048.31—
SDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.048.2735.76
SpecCascadeTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.048.2739.42
SD (Lossy)Target/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.047.9738.83
SD (Lossy)Target/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.047.9640.91
Static EnsembleTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.047.956.31
Static EnsembleTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.047.956.31
Static EnsembleTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.047.961.42
SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.047.8237.67
DiversedTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.047.6694.81
SpecCascadeTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.047.6143.95
AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.047.56—
SpecCascadeTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.047.5340.22
Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.047.5361.44
DIVERSEDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.047.5264.75
Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.047.3151.2
DIVERSEDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.047.2370.53
DiversedTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.047.2263.38
DiversedTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.047.2263.38
Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.047.2251.58
DIVERSEDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.047.2263.38
SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.047.0920.44
AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.047.03—
DiversedTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.045.5438.12
Static EnsembleTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.045.3934.14
AutoregressiveTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.045.38—
SDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.045.3519.56
SDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.045.2616.45
DIVERSEDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.045.2545.87
AutoregressiveTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.045.21—
Static EnsembleTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.045.1930.8
DIVERSEDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.045.0147.01
Static EnsembleTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.044.9821.24
SpecCascadeTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.044.9719.45
SDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.044.9515.6
SpecCascadeTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.044.9132.98
SD (lossy)Target Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.044.919.56
AutoregressiveTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.044.9—
SD (lossy)Target/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.044.919.56