Throughput and Latency Evaluation on MT-Bench
24.1Throughput (TPS)MoE-SpAc
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MoE-SpAc2026.02 | 24.1 | 22.26 | |
| llama.cppspeculative decoding=true2026.02 | 17.58 | 29.03 | |
| llama.cppspeculative decoding=false2026.02 | 14.34 | 36.08 | |
| HybriMoE2026.02 | 12.03 | 43.47 | |
| Fate2026.02 | 7.89 | 66.89 | |
| SP-MoE2026.02 | 4.46 | 111.34 | |
| MoE-Infinity2026.02 | 3.64 | 136.84 | |
| vLLM2026.02 | 2.67 | 187.88 | |
| Mixtral Offload2026.02 | 2.36 | 212.7 | |
| Accelerate2026.02 | 1.15 | 435.55 |