Generative Inference on MT-Bench
2.73SpeedupDART
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| DARTModel=Qwen4B, Temperature=02026.01 | 2.73 | — | |
| DARTModel=L2 7B, Temperature=02026.01 | 2.61 | — | |
| SpecBundleTarget Model=Llama-4-Scout, Draft Model=SpecBundle, #GPUs=82026.03 | 2.61 | 1,312.4 | |
| DARTModel=Qwen1.7B, Temperature=02026.01 | 2.57 | — | |
| EAGLE3Target Model=Llama-4-Scout, Draft Model=Existing, #GPUs=82026.03 | 2.5 | 1,253 | |
| HydraModel=L2 7B, Temperature=02026.01 | 2.48 | — | |
| EAGLE3Target Model=Llama-3.1-8B, Draft Model=Existing, #GPUs=12026.03 | 2.39 | 454.7 | |
| SpecBundleTarget Model=Llama-3.1-8B, Draft Model=SpecBundle, #GPUs=12026.03 | 2.37 | 450 | |
| EAGLE3Target Model=Llama-3.3-70B, Draft Model=Existing, #GPUs=42026.03 | 2.35 | 1,272.7 | |
| SpecBundleTarget Model=Llama-3.3-70B, Draft Model=SpecBundle, #GPUs=42026.03 | 2.31 | 1,253 | |
| SPACEBackbone=LLaMA-2-70B2024.02 | 2.26 | — | |
| DARTModel=Qwen32B, Temperature=02026.01 | 2.24 | — | |
| DARTModel=Qwen8B, Temperature=02026.01 | 2.22 | — | |
| DARTModel=Qwen14B, Temperature=02026.01 | 2.2 | — | |
| EAGLE3Model=Qwen4B, Temperature=02026.01 | 2.17 | — | |
| MedusaModel=L2 7B, Temperature=02026.01 | 2.12 | — | |
| EAGLE3Model=Qwen8B, Temperature=02026.01 | 2.08 | — | |
| EAGLE3Model=Qwen1.7B, Temperature=02026.01 | 1.98 | — | |
| DARTModel=Qwen14B, Temperature=12026.01 | 1.94 | — | |
| DARTModel=Qwen32B, Temperature=12026.01 | 1.82 | — | |
| SpecDecBackbone=LLaMA-2-70B2024.02 | 1.79 | — | |
| EAGLE3Model=Qwen32B, Temperature=02026.01 | 1.76 | — | |
| EAGLE3Model=Qwen14B, Temperature=02026.01 | 1.71 | — | |
| LookaheadModel=L2 7B, Temperature=02026.01 | 1.63 | — | |
| EAGLE3Model=Qwen14B, Temperature=12026.01 | 1.61 | — | |
| EAGLE3Model=Qwen32B, Temperature=12026.01 | 1.58 | — | |
| PLDModel=L2 7B, Temperature=02026.01 | 1.57 | — | |
| SpecBundleTarget Model=Qwen-30B-A3B, Draft Model=SpecBundle, #GPUs=42026.03 | 1.55 | 2,086.1 | |
| SpecBundleTarget Model=Qwen-235B-A22B, Draft Model=SpecBundle, #GPUs=82026.03 | 1.54 | 814.5 | |
| HF AssistGenBackbone=LLaMA-2-70B2024.02 | 1.53 | — | |
| SpecBundleTarget Model=Ling-Flash-V2, Draft Model=SpecBundle, #GPUs=82026.03 | 1.4 | 1,022.6 | |
| SpecBundleTarget Model=Kimi-K2, Draft Model=SpecBundle, #GPUs=82026.03 | 1.24 | 533.8 | |
| Look-aheadBackbone=LLaMA-2-70B2024.02 | 1.22 | — | |
| EAGLE3Target Model=Qwen-235B-A22B, Draft Model=Existing, #GPUs=82026.03 | 1.21 | 642.7 | |
| Self-SpecDecBackbone=LLaMA-2-70B2024.02 | 1.15 | — | |
| SPSModel=Qwen32B, Temperature=02026.01 | 1.12 | — | |
| Standard InferenceTarget Model=Llama-3.1-8B, Draft Model=-, #GPUs=12026.03 | 1 | 190 | |
| Standard InferenceTarget Model=Llama-3.3-70B, Draft Model=-, #GPUs=42026.03 | 1 | 540.5 | |
| Standard InferenceTarget Model=Llama-4-Scout, Draft Model=-, #GPUs=82026.03 | 1 | 502.1 | |
| Standard InferenceTarget Model=Qwen-30B-A3B, Draft Model=-, #GPUs=42026.03 | 1 | 1,341.3 | |
| Standard InferenceTarget Model=Qwen-235B-A22B, Draft Model=-, #GPUs=82026.03 | 1 | 529.9 | |
| Standard InferenceTarget Model=Ling-Flash-V2, Draft Model=-, #GPUs=82026.03 | 1 | 728.5 | |
| Standard InferenceTarget Model=Kimi-K2, Draft Model=-, #GPUs=82026.03 | 1 | 430.9 | |
| SPSModel=Qwen14B, Temperature=02026.01 | 0.92 | — |