List your company
ResearchFormerly WizWand

End-to-End Inference on LMSys-Chat-1M and ShareGPT traces

74.46p99 TTFT (ms)Sem-MoE
72.64884.87997.11109.341Mar 6, 2025
Updated 8mo ago

Evaluation results

MethodLinks
Sem-MoEModel=Qwen3-30B-A3B, Input Length=256, Scenario=Attention-TP2025.0374.46750.26
Sem-MoEModel=DeepSeek-V2-Lite, Input Length=256, Scenario=Attention-TP2025.0375.63603.1
MoETunerModel=DeepSeek-V2-Lite, Input Length=256, Scenario=Attention-TP2025.0380.96604.46
Sem-MoEModel=Qwen3-30B-A3B, Input Length=512, Scenario=Attention-TP2025.0383.87759.16
SGLangModel=DeepSeek-V2-Lite, Input Length=256, Scenario=Attention-TP2025.0384.87617.87
SGLangModel=Qwen3-30B-A3B, Input Length=256, Scenario=Attention-TP2025.0385.72758.6
MoETunerModel=Qwen3-30B-A3B, Input Length=256, Scenario=Attention-TP2025.0387.24763.97
Sem-MoEModel=DeepSeek-V2-Lite, Input Length=512, Scenario=Attention-TP2025.0388.69599.89
MoETunerModel=DeepSeek-V2-Lite, Input Length=512, Scenario=Attention-TP2025.0392.17602.02
SGLangModel=DeepSeek-V2-Lite, Input Length=512, Scenario=Attention-TP2025.0398.1609.81
Sem-MoEModel=DeepSeek-V2-Lite, Input Length=1024, Scenario=Attention-TP2025.03100.74604.45
MoETunerModel=Qwen3-30B-A3B, Input Length=512, Scenario=Attention-TP2025.03101.86759.41
Sem-MoEModel=Qwen3-30B-A3B, Input Length=1024, Scenario=Attention-TP2025.03103.69762.3
SGLangModel=Qwen3-30B-A3B, Input Length=512, Scenario=Attention-TP2025.03104.76766.8
MoETunerModel=Qwen3-30B-A3B, Input Length=1024, Scenario=Attention-TP2025.03107.36764.16
SGLangModel=Qwen3-30B-A3B, Input Length=1024, Scenario=Attention-TP2025.03107.73769.83
SGLangModel=DeepSeek-V2-Lite, Input Length=1024, Scenario=Attention-TP2025.03111.19608.96
MoETunerModel=DeepSeek-V2-Lite, Input Length=1024, Scenario=Attention-TP2025.03119.76606.54