Inference Latency and Throughput on CNN/DM (Summarization)
3.58SpeedupTALON
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| TALONModel=Vicuna-13B, Temperature=02026.01 | 3.58 | — | 7.2 | |
| EAGLE-3Model=Vicuna-13B, Temperature=02026.01 | 3.43 | — | 6.93 | |
| TALONModel=Llama3-8B, Temperature=02026.01 | 3.3 | — | 5.28 | |
| OPT-TreeModel=Vicuna-13B, Temperature=02026.01 | 3.22 | — | 6.91 | |
| TALONModel=DSL-8B, Temperature=02026.01 | 3.14 | — | 4.75 | |
| EAGLE-3Model=Llama3-8B, Temperature=02026.01 | 3.06 | — | 5.46 | |
| EAGLE-3Model=DSL-8B, Temperature=02026.01 | 2.89 | — | 5.03 | |
| TALONModel=Qwen3-8B, Temperature=02026.01 | 2.3 | — | 3.2 | |
| EAGLE-3Model=Qwen3-8B, Temperature=02026.01 | 1.95 | — | 3.27 | |
| HYDRAModel=Vicuna-13B, Temperature=02026.01 | 1.86 | — | 2.82 | |
| TALONModel=Qwen3-32B, Temperature=02026.01 | 1.72 | — | 2.54 | |
| PACERLLM Family/Size=Llama-2 7B/70B2026.02 | 1.71 | 14.5 | — | |
| SpeculativeLLM Family/Size=Llama-2 7B/70B2026.02 | 1.7 | 14.4 | — | |
| EAGLE-3Model=Qwen3-32B, Temperature=02026.01 | 1.57 | — | 2.57 | |
| MEDUSAModel=Vicuna-13B, Temperature=02026.01 | 1.56 | — | 2.09 | |
| SDModel=Vicuna-13B, Temperature=02026.01 | 1.41 | — | 2.63 | |
| PACERLLM Family/Size=Deepseek 6.7B/33B2026.02 | 1.4 | 21.51 | — | |
| PACERLLM Family/Size=Deepseek 1.3B/33B2026.02 | 1.39 | 23.77 | — | |
| SpeculativeLLM Family/Size=Deepseek 1.3B/33B2026.02 | 1.38 | 23.55 | — | |
| SpeculativeLLM Family/Size=Deepseek 6.7B/33B2026.02 | 1.34 | 20.55 | — | |
| AssistLLM Family/Size=Llama-2 7B/70B2026.02 | 1.34 | 11.36 | — | |
| AssistLLM Family/Size=Deepseek 1.3B/33B2026.02 | 1.3 | 22.27 | — | |
| AssistLLM Family/Size=Deepseek 6.7B/33B2026.02 | 1.2 | 18.52 | — | |
| LookaheadLLM Family/Size=Deepseek 6.7B/33B2026.02 | 1.15 | 17.64 | — | |
| RESTLLM Family/Size=Llama-2 7B/70B2026.02 | 1.14 | 9.62 | — | |
| RESTLLM Family/Size=Deepseek 6.7B/33B2026.02 | 1.09 | 16.7 | — | |
| LookaheadLLM Family/Size=Deepseek 1.3B/33B2026.02 | 1.08 | 18.21 | — | |
| RESTLLM Family/Size=Deepseek 1.3B/33B2026.02 | 1.06 | 18.5 | — | |
| LookaheadLLM Family/Size=Llama-2 7B/70B2026.02 | 1.01 | 8.56 | — | |
| VanillaLLM Family/Size=Deepseek 1.3B/33B2026.02 | 1 | 17.11 | — | |
| VanillaLLM Family/Size=Deepseek 6.7B/33B2026.02 | 1 | 15.37 | — | |
| VanillaLLM Family/Size=Llama-2 7B/70B2026.02 | 1 | 8.46 | — | |
| DDDTemperature=0, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=4, steps=4, topk=8, draft_tokens=322026.05 | — | 180.66 | — | |
| DDDTemperature=1, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=4, steps=4, topk=8, draft_tokens=322026.05 | — | 174.18 | — | |
| EAGLE-3Temperature=0, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=4, steps=4, topk=8, draft_tokens=322026.05 | — | 193.6 | — | |
| EAGLE-3Temperature=1, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=4, steps=4, topk=8, draft_tokens=322026.05 | — | 185.41 | — | |
| EVICTTemperature=0, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=4, steps=4, topk=8, draft_tokens=322026.05 | — | 216.51 | — | |
| EVICTTemperature=1, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=4, steps=4, topk=8, draft_tokens=322026.05 | — | 208.19 | — | |
| LookaheadTemperature=0, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=42026.05 | — | 104.45 | — | |
| LookaheadTemperature=1, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=42026.05 | — | 102.95 | — | |
| VanillaTemperature=0, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=42026.05 | — | 174.37 | — | |
| VanillaTemperature=1, Backbone=Ling-flash-2.0, Batch Size=1, TP (Tensor Parallelism)=42026.05 | — | 170.29 | — |