List your company
ResearchFormerly WizWand

Chatbot Workload on LMSYS-Chat-1M

0.47Average PTLA (s/token)TIE
0.02683.01846.019.0016Apr 1, 2026
Updated 7mo ago

Evaluation results

MethodLinks
TIEModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.040.4793.04
SSJFModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.040.69106.97
TIEModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.040.7159.05
TIEModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.040.866.51
SSJFModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.041.0378.15
LTRModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.041.2588.21
LTRModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.041.34113.38
TIEModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.46729.42
LTRModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.041.4685.02
TIEModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.5973.62
SSJFModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.041.67100.72
SSJFModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.86855.83
FCFSModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.041.91101.86
LTRModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.98852.08
FCFSModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.042.56127.67
FCFSModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.042.65163.53
FCFSModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.042.711,057.02
TIEModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.042.86109.97
LTRModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.042.9596.26
TIEModel=GPT-oss-20B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.043112.34
SSJFModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.043.4109.49
SSJFModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.043.53141.52
LTRModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.044.51141.35
LTRModel=GPT-oss-20B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.044.96149.46
SSJFModel=GPT-oss-20B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.046.77167.68
FCFSModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.048.07150.18
FCFSModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.049.6201.17
FCFSModel=GPT-oss-20B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.0411.55188.33