End-to-End Inference on LMSys-Chat-1M and ShareGPT traces
74.46p99 TTFT (ms)Sem-MoE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Sem-MoEModel=Qwen3-30B-A3B, Input Length=256, Scenario=Attention-TP2025.03 | 74.46 | 750.26 | |
| Sem-MoEModel=DeepSeek-V2-Lite, Input Length=256, Scenario=Attention-TP2025.03 | 75.63 | 603.1 | |
| MoETunerModel=DeepSeek-V2-Lite, Input Length=256, Scenario=Attention-TP2025.03 | 80.96 | 604.46 | |
| Sem-MoEModel=Qwen3-30B-A3B, Input Length=512, Scenario=Attention-TP2025.03 | 83.87 | 759.16 | |
| SGLangModel=DeepSeek-V2-Lite, Input Length=256, Scenario=Attention-TP2025.03 | 84.87 | 617.87 | |
| SGLangModel=Qwen3-30B-A3B, Input Length=256, Scenario=Attention-TP2025.03 | 85.72 | 758.6 | |
| MoETunerModel=Qwen3-30B-A3B, Input Length=256, Scenario=Attention-TP2025.03 | 87.24 | 763.97 | |
| Sem-MoEModel=DeepSeek-V2-Lite, Input Length=512, Scenario=Attention-TP2025.03 | 88.69 | 599.89 | |
| MoETunerModel=DeepSeek-V2-Lite, Input Length=512, Scenario=Attention-TP2025.03 | 92.17 | 602.02 | |
| SGLangModel=DeepSeek-V2-Lite, Input Length=512, Scenario=Attention-TP2025.03 | 98.1 | 609.81 | |
| Sem-MoEModel=DeepSeek-V2-Lite, Input Length=1024, Scenario=Attention-TP2025.03 | 100.74 | 604.45 | |
| MoETunerModel=Qwen3-30B-A3B, Input Length=512, Scenario=Attention-TP2025.03 | 101.86 | 759.41 | |
| Sem-MoEModel=Qwen3-30B-A3B, Input Length=1024, Scenario=Attention-TP2025.03 | 103.69 | 762.3 | |
| SGLangModel=Qwen3-30B-A3B, Input Length=512, Scenario=Attention-TP2025.03 | 104.76 | 766.8 | |
| MoETunerModel=Qwen3-30B-A3B, Input Length=1024, Scenario=Attention-TP2025.03 | 107.36 | 764.16 | |
| SGLangModel=Qwen3-30B-A3B, Input Length=1024, Scenario=Attention-TP2025.03 | 107.73 | 769.83 | |
| SGLangModel=DeepSeek-V2-Lite, Input Length=1024, Scenario=Attention-TP2025.03 | 111.19 | 608.96 | |
| MoETunerModel=DeepSeek-V2-Lite, Input Length=1024, Scenario=Attention-TP2025.03 | 119.76 | 606.54 |