List your company
ResearchFormerly WizWand

Generative Inference on MT-Bench

2.73SpeedupDART
0.84761.33631.8252.3137Feb 19, 2024Jun 24, 2024Oct 29, 2024Mar 4, 2025Jul 9, 2025Nov 12, 2025Mar 19, 2026
Updated 7mo ago

Evaluation results

MethodLinks
DARTModel=Qwen4B, Temperature=02026.012.73—
DARTModel=L2 7B, Temperature=02026.012.61—
SpecBundleTarget Model=Llama-4-Scout, Draft Model=SpecBundle, #GPUs=82026.032.611,312.4
DARTModel=Qwen1.7B, Temperature=02026.012.57—
EAGLE3Target Model=Llama-4-Scout, Draft Model=Existing, #GPUs=82026.032.51,253
HydraModel=L2 7B, Temperature=02026.012.48—
EAGLE3Target Model=Llama-3.1-8B, Draft Model=Existing, #GPUs=12026.032.39454.7
SpecBundleTarget Model=Llama-3.1-8B, Draft Model=SpecBundle, #GPUs=12026.032.37450
EAGLE3Target Model=Llama-3.3-70B, Draft Model=Existing, #GPUs=42026.032.351,272.7
SpecBundleTarget Model=Llama-3.3-70B, Draft Model=SpecBundle, #GPUs=42026.032.311,253
SPACEBackbone=LLaMA-2-70B2024.022.26—
DARTModel=Qwen32B, Temperature=02026.012.24—
DARTModel=Qwen8B, Temperature=02026.012.22—
DARTModel=Qwen14B, Temperature=02026.012.2—
EAGLE3Model=Qwen4B, Temperature=02026.012.17—
MedusaModel=L2 7B, Temperature=02026.012.12—
EAGLE3Model=Qwen8B, Temperature=02026.012.08—
EAGLE3Model=Qwen1.7B, Temperature=02026.011.98—
DARTModel=Qwen14B, Temperature=12026.011.94—
DARTModel=Qwen32B, Temperature=12026.011.82—
SpecDecBackbone=LLaMA-2-70B2024.021.79—
EAGLE3Model=Qwen32B, Temperature=02026.011.76—
EAGLE3Model=Qwen14B, Temperature=02026.011.71—
LookaheadModel=L2 7B, Temperature=02026.011.63—
EAGLE3Model=Qwen14B, Temperature=12026.011.61—
EAGLE3Model=Qwen32B, Temperature=12026.011.58—
PLDModel=L2 7B, Temperature=02026.011.57—
SpecBundleTarget Model=Qwen-30B-A3B, Draft Model=SpecBundle, #GPUs=42026.031.552,086.1
SpecBundleTarget Model=Qwen-235B-A22B, Draft Model=SpecBundle, #GPUs=82026.031.54814.5
HF AssistGenBackbone=LLaMA-2-70B2024.021.53—
SpecBundleTarget Model=Ling-Flash-V2, Draft Model=SpecBundle, #GPUs=82026.031.41,022.6
SpecBundleTarget Model=Kimi-K2, Draft Model=SpecBundle, #GPUs=82026.031.24533.8
Look-aheadBackbone=LLaMA-2-70B2024.021.22—
EAGLE3Target Model=Qwen-235B-A22B, Draft Model=Existing, #GPUs=82026.031.21642.7
Self-SpecDecBackbone=LLaMA-2-70B2024.021.15—
SPSModel=Qwen32B, Temperature=02026.011.12—
Standard InferenceTarget Model=Llama-3.1-8B, Draft Model=-, #GPUs=12026.031190
Standard InferenceTarget Model=Llama-3.3-70B, Draft Model=-, #GPUs=42026.031540.5
Standard InferenceTarget Model=Llama-4-Scout, Draft Model=-, #GPUs=82026.031502.1
Standard InferenceTarget Model=Qwen-30B-A3B, Draft Model=-, #GPUs=42026.0311,341.3
Standard InferenceTarget Model=Qwen-235B-A22B, Draft Model=-, #GPUs=82026.031529.9
Standard InferenceTarget Model=Ling-Flash-V2, Draft Model=-, #GPUs=82026.031728.5
Standard InferenceTarget Model=Kimi-K2, Draft Model=-, #GPUs=82026.031430.9
SPSModel=Qwen14B, Temperature=02026.010.92—