Chatbot Workload on LMSYS-Chat-1M
0.47Average PTLA (s/token)TIE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| TIEModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.04 | 0.47 | 93.04 | |
| SSJFModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.04 | 0.69 | 106.97 | |
| TIEModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.04 | 0.71 | 59.05 | |
| TIEModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.04 | 0.8 | 66.51 | |
| SSJFModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.04 | 1.03 | 78.15 | |
| LTRModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.04 | 1.25 | 88.21 | |
| LTRModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.04 | 1.34 | 113.38 | |
| TIEModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.46 | 729.42 | |
| LTRModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.04 | 1.46 | 85.02 | |
| TIEModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.59 | 73.62 | |
| SSJFModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.04 | 1.67 | 100.72 | |
| SSJFModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.86 | 855.83 | |
| FCFSModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.04 | 1.91 | 101.86 | |
| LTRModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.98 | 852.08 | |
| FCFSModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.04 | 2.56 | 127.67 | |
| FCFSModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.04 | 2.65 | 163.53 | |
| FCFSModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 2.71 | 1,057.02 | |
| TIEModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.04 | 2.86 | 109.97 | |
| LTRModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 2.95 | 96.26 | |
| TIEModel=GPT-oss-20B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 3 | 112.34 | |
| SSJFModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 3.4 | 109.49 | |
| SSJFModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.04 | 3.53 | 141.52 | |
| LTRModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.04 | 4.51 | 141.35 | |
| LTRModel=GPT-oss-20B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 4.96 | 149.46 | |
| SSJFModel=GPT-oss-20B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 6.77 | 167.68 | |
| FCFSModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 8.07 | 150.18 | |
| FCFSModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.04 | 9.6 | 201.17 | |
| FCFSModel=GPT-oss-20B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 11.55 | 188.33 |