List your company
ResearchFormerly WizWand

Chatbot Latency on Alpaca

0.3Average PTLA (s/token)TIE
0.15561.13032.1053.0797Apr 1, 2026
Updated 7mo ago

Evaluation results

MethodLinks
TIEModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.040.363.47
TIEModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.040.4745.04
SSJFModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.040.4772.27
TIEModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.040.4929.88
TIEModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.040.5142.9
LTRModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.040.5574.92
LTRModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.040.7159.28
LTRModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.040.8164.18
LTRModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.040.8235.55
SSJFModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.040.8863.83
SSJFModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.040.9467.72
SSJFModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.1440.82
TIEModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.041.1867.77
TIEModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.25625.56
TIEModel=GPT-oss-20B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.2671.79
FCFSModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.041.3176.27
FCFSModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.041.3486.13
LTRModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.041.4680.26
FCFSModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.041.46109.42
LTRModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.52743.32
SSJFModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.68777.19
LTRModel=GPT-oss-20B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.042.0697.66
FCFSModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.042.13890.31
SSJFModel=GPT-oss-20B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.042.1998.59
SSJFModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.042.2493.37
FCFSModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.042.5859.72
FCFSModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.043.62111.22
FCFSModel=GPT-oss-20B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.043.91117.35