List your company
ResearchFormerly WizWand

Chatbot Workload on ShareGPT

0.36Average PTLA (s/token)TIE
0.21.282.363.44Apr 1, 2026
Updated 7mo ago

Evaluation results

MethodLinks
TIEModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.040.36190.33——
TIEModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.040.5797.71——
SSJFModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.040.57213.25——
TIETesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.040.5798.10.9239.04
TIEModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.040.694.13——
SSJFModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.040.75114.01——
LTRModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.040.75212.58——
LTRModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.040.79131.63——
SSJFModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.040.82143.29——
TIEModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.040.86116.26——
LTRTesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.040.86120.031.56273.87
LTRModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.040.9117.99——
SSJFTesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.040.95132.581.84296.69
TIEModel=GPT-oss-20B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.1171.65——
FCFSModel=Mistral-7B-Instruct-v0.3, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=12026.041.19174.43——
TIEModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.041.23183.12——
FCFSModel=Mixtral-8x7B-Instruct-v0.1, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=42026.041.24137.17——
TIETesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.041.41288.792.22660.28
LTRModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.5150.54——
TIEModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=TIE, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.52994.56——
FCFSModel=OpenPanGu-7B, Workload=Chatbot, Request Rate=100 RPS2026.041.64270.92——
FCFSTesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.041.66161.563.27316.07
SSJFModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.75172.47——
LTRModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.821,268.46——
SSJFModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.871,280.86——
LTRModel=GPT-oss-20B, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.041.97214.05——
LTRModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=LTR, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.041.97218.3——
SSJFModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.042.17235.27——
LTRTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.042.22346.53.78696.75
FCFSModel=DeepSeek-R1-Distill-Qwen-32B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.042.421,443.49——
SSJFTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.042.43342.454.9744.59
SSJFModel=GPT-oss-20B, Scheduling Strategy=SSJF, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.043.03244.73——
FCFSModel=Qwen3-30B-A3B-Instruct-2507, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.043.76224.7——
FCFSModel=GPT-oss-20B, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=22026.043.83255.15——
FCFSModel=Qwen3-Next-80B-A3B-Instruct, Scheduling Strategy=FCFS, Requests Per Second (RPS)=100, Precision=FP16, Number of GPUs=82026.044.19285.93——
FCFSTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.044.36417.288.78807.27