LLM Inference Scheduling on LMSYS-Chat-1M
0.1432Average Per-token Latency (s/token)Oracle SJF
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| Oracle SJFModel=Llama2025.09 | 0.1432 | — | — | — | |
| Oracle SJFModel=DeepSeek-R12025.09 | 0.1532 | — | — | — | |
| PARSModel=Llama2025.09 | 0.1864 | — | — | — | |
| Cross-Model PARSModel=Llama2025.09 | 0.1879 | — | — | — | |
| PARSModel=DeepSeek-R12025.09 | 0.2069 | — | — | — | |
| Listwise SJFModel=Llama2025.09 | 0.2146 | — | — | — | |
| Cross-Model PARSModel=DeepSeek-R12025.09 | 0.2559 | — | — | — | |
| Listwise SJFModel=DeepSeek-R12025.09 | 0.2749 | — | — | — | |
| Pointwise SJFModel=DeepSeek-R12025.09 | 0.3217 | — | — | — | |
| Pointwise SJFModel=Llama2025.09 | 0.3427 | — | — | — | |
| FCFSModel=DeepSeek-R12025.09 | 0.521 | — | — | — | |
| FCFSModel=Llama2025.09 | 1.2699 | — | — | — | |
| TIETesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 2.41 | 4.05 | 204.03 | 475.1 | |
| LTRTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 4.34 | 7.03 | 252.2 | 507.35 | |
| SSJFTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 5.5 | 8.24 | 273.3 | 551.15 | |
| FCFSTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 9.08 | 16.13 | 319.51 | 618.21 | |
| Cross-Model PARSModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | — | 0.5485 | — | — | |
| Cross-Model PARSModel=Llama, Workload Mode=Burst Mode2025.09 | — | 0.2621 | — | — | |
| FCFSModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | — | 1.0911 | — | — | |
| FCFSModel=Llama, Workload Mode=Burst Mode2025.09 | — | 2.3075 | — | — | |
| Listwise SJFModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | — | 0.5855 | — | — | |
| Listwise SJFModel=Llama, Workload Mode=Burst Mode2025.09 | — | 0.3061 | — | — | |
| Oracle SJFModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | — | 0.259 | — | — | |
| Oracle SJFModel=Llama, Workload Mode=Burst Mode2025.09 | — | 0.2088 | — | — | |
| PARSModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | — | 0.4226 | — | — | |
| PARSModel=Llama, Workload Mode=Burst Mode2025.09 | — | 0.261 | — | — | |
| Pointwise SJFModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | — | 0.6668 | — | — | |
| Pointwise SJFModel=Llama, Workload Mode=Burst Mode2025.09 | — | 0.5535 | — | — |