Text Generation Speed/Throughput on MT-Bench
4.76SpeedupHybrid Verified Decoding
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Hybrid Verified DecodingTarget Model=Llama3.1-8B, Reference Baseline=SuffixDecoding2026.05 | 4.76 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-8B, Reference Baseline=SuffixDecoding2026.05 | 4.68 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-8B, Reference Baseline=Greedy Decoding2026.05 | 3.34 | — | |
| Hybrid Verified DecodingTarget Model=Llama3.1-8B, Reference Baseline=Greedy Decoding2026.05 | 3.24 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-8B, Serving Stack=vLLM, Baseline Comparison=vs. EAGLE32026.05 | 2.31 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-4B, Reference Baseline=SuffixDecoding2026.05 | 2.06 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-4B, Reference Baseline=Greedy Decoding2026.05 | 1.59 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-8B, Reference Baseline=Frequency-Based Hybrid Decoding2026.05 | 1.41 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-8B, Serving Stack=vLLM, Baseline Comparison=vs. SuffixDecoding2026.05 | 1.21 | — | |
| Hybrid Verified DecodingTarget Model=Llama3.1-8B, Reference Baseline=EAGLE32026.05 | 1.18 | — | |
| Hybrid Verified DecodingTarget Model=Llama3.1-8B, Reference Baseline=Frequency-Based Hybrid Decoding2026.05 | 1.1 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-4B, Reference Baseline=EAGLE32026.05 | 0.99 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-8B, Reference Baseline=EAGLE32026.05 | 0.98 | — | |
| Hybrid Verified DecodingTarget Model=Qwen3-4B, Reference Baseline=Frequency-Based Hybrid Decoding2026.05 | 0.87 | — | |
| DDDTemperature=0, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 107.17 | |
| DDDTemperature=1, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 105.36 | |
| EAGLE-3Temperature=0, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 112.68 | |
| EAGLE-3Temperature=1, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 108.28 | |
| EVICTTemperature=0, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 137.81 | |
| EVICTTemperature=1, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 134.18 | |
| LookaheadTemperature=0, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 64.73 | |
| LookaheadTemperature=1, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 63.5 | |
| VanillaTemperature=0, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 80.68 | |
| VanillaTemperature=1, Backbone=Qwen3-235B-A22B, Inference Engine=SGLang, Hardware=NVIDIA A100 SXM4 80GB GPUs, Batch Size=1, Tensor Parallelism=TP=82026.05 | — | 80.99 |