Chatbot Workload on ShareGPT
0.36Average PTLA (s/token)TIE
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| TIEModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.04 | 0.36 | 190.33 | — | — | |
| TIEModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.04 | 0.57 | 97.71 | — | — | |
| SSJFModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.04 | 0.57 | 213.25 | — | — | |
| TIETesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 0.57 | 98.1 | 0.9 | 239.04 | |
| TIEModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.04 | 0.6 | 94.13 | — | — | |
| SSJFModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.04 | 0.75 | 114.01 | — | — | |
| LTRModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.04 | 0.75 | 212.58 | — | — | |
| LTRModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.04 | 0.79 | 131.63 | — | — | |
| SSJFModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.04 | 0.82 | 143.29 | — | — | |
| TIEModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 0.86 | 116.26 | — | — | |
| LTRTesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 0.86 | 120.03 | 1.56 | 273.87 | |
| LTRModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.04 | 0.9 | 117.99 | — | — | |
| SSJFTesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 0.95 | 132.58 | 1.84 | 296.69 | |
| TIEModel=GPT-oss-20B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.1 | 171.65 | — | — | |
| FCFSModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.04 | 1.19 | 174.43 | — | — | |
| TIEModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.04 | 1.23 | 183.12 | — | — | |
| FCFSModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.04 | 1.24 | 137.17 | — | — | |
| TIETesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 1.41 | 288.79 | 2.22 | 660.28 | |
| LTRModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.5 | 150.54 | — | — | |
| TIEModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.52 | 994.56 | — | — | |
| FCFSModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.04 | 1.64 | 270.92 | — | — | |
| FCFSTesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 1.66 | 161.56 | 3.27 | 316.07 | |
| SSJFModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.75 | 172.47 | — | — | |
| LTRModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.82 | 1,268.46 | — | — | |
| SSJFModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.87 | 1,280.86 | — | — | |
| LTRModel=GPT-oss-20B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 1.97 | 214.05 | — | — | |
| LTRModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.04 | 1.97 | 218.3 | — | — | |
| SSJFModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.04 | 2.17 | 235.27 | — | — | |
| LTRTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 2.22 | 346.5 | 3.78 | 696.75 | |
| FCFSModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 2.42 | 1,443.49 | — | — | |
| SSJFTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 2.43 | 342.45 | 4.9 | 744.59 | |
| SSJFModel=GPT-oss-20B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 3.03 | 244.73 | — | — | |
| FCFSModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 3.76 | 224.7 | — | — | |
| FCFSModel=GPT-oss-20B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.04 | 3.83 | 255.15 | — | — | |
| FCFSModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.04 | 4.19 | 285.93 | — | — | |
| FCFSTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 4.36 | 417.28 | 8.78 | 807.27 |