List your company
ResearchFormerly WizWand

Summarization on CNNDM (Accept Rate, ROUGE-2)

12.64ROUGE-2Diversed
8.85449.837210.8211.8028Apr 8, 2026
Updated 6mo ago

Evaluation results

MethodLinks
DiversedTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.0412.6458.46
DiversedTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0, Draft length (N)=52026.0412.6458.46
DiversedTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.0412.3781.92
DIVERSEDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.0412.1169.96
Static EnsembleTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.0411.8541.6
Static EnsembleTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0, Draft length (N)=52026.0411.8541.6
SpecCascadeTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.0411.7447.29
DIVERSEDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.0411.650.9211
DiversedTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.0411.664.26
AutoregressiveTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.0411.48—
AutoregressiveTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0, Draft length (N)=52026.0411.48—
DiversedTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.0411.4766.39
DiversedTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.0411.4766.39
Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.0411.4661.06
Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.0411.4585.04
SDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=0, Draft length (N)=52026.0411.3836.4
SDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0, Draft length (N)=52026.0411.3836.4
DIVERSEDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.0411.3746.4
SpecCascadeTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.0411.337.67
AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.0411.29—
SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0, Draft length (N)=52026.0411.2671.7
AutoregressiveTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.0411.02—
DIVERSEDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.0410.9746.59
Static EnsembleTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.0410.940.67
Static EnsembleTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.0410.8841.07
AutoregressiveTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.0410.87—
AutoregressiveTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.0410.87—
DIVERSEDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.0410.8666.9
DIVERSEDTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.0410.8666.9
SpecCascadeTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.0410.8538.13
Static EnsembleTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.0410.8454.48
Static EnsembleTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.0410.8454.48
Static EnsembleTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.0410.8454.48
Static EnsembleTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.0410.8454.48
SD (lossy)Target Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.0410.7435.32
Static EnsembleTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.0410.7447.47
SpecCascadeTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.0410.7248.72
SD (Lossy)Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.0410.7148.73
SDTemperature=0, Draft length (N)=5, Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B2026.0410.6240.39
SDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=0, Draft length (N)=52026.0410.6240.39
SD (Lossy)Target/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.0410.6245.43
SD (Lossy)Target Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.0410.6245.43
SD (lossy)Target/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.0410.5336.4
SD (Lossy)Target/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.0410.5138.86
Static EnsembleTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.0410.460.8758
SpecCascadeTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.0410.4251.44
SpecCascadeTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.0410.4251.44
SDTarget Model=Qwen3-8B, Draft Model=Qwen3-0.6B, Temperature=0.1, Draft Length=52026.0410.3534.19
SpecCascadeTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.0410.190.522
SD (Lossy)Target/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.0410.170.1785
AutoregressiveTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.049.97—
AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.049.94—
SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=0.1, Draft length (N)=52026.049.920.1769
AutoregressiveTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.049.86—
SDTarget/Draft model pair=Qwen3-8B/Qwen3-0.6B, Temperature=1, Draft length (N)=52026.049.8535.26
SDTarget/Draft model pair=Llama-3.1-8B/Llama-3.2-1B, Temperature=1, Draft length (N)=52026.049.4621.6
SDTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.049.0640.39
SDTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.049.0640.39
AutoregressiveTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.049.01—
AutoregressiveTarget/Draft model pair=Gemma-3-12B/Gemma-3-4B, Temperature=1, Draft length (N)=52026.049.01—
AutoregressiveTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=1, Draft Length (N)=52026.049.01—
SDTarget Model=Gemma-3-12B, Draft Model=Gemma-3-4B, Temperature=0.1, Draft Length (N)=52026.04944.27