LLM Inference Scheduling on Alpaca
0.1576P90 Per-token Latency (s/token)Oracle SJF
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Oracle SJFModel=Llama, Workload Mode=Burst Mode2025.09 | 0.1576 | — | — | — | |
| PARSModel=Llama, Workload Mode=Burst Mode2025.09 | 0.18 | — | — | — | |
| Cross-Model PARSModel=Llama, Workload Mode=Burst Mode2025.09 | 0.208 | — | — | — | |
| Listwise SJFModel=Llama, Workload Mode=Burst Mode2025.09 | 0.2155 | — | — | — | |
| Oracle SJFModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | 0.2298 | — | — | — | |
| Pointwise SJFModel=Llama, Workload Mode=Burst Mode2025.09 | 0.2306 | — | — | — | |
| PARSModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | 0.2581 | — | — | — | |
| Cross-Model PARSModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | 0.3334 | — | — | — | |
| Listwise SJFModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | 0.3868 | — | — | — | |
| Pointwise SJFModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | 0.7052 | — | — | — | |
| FCFSModel=DeepSeek-R1, Workload Mode=Burst Mode2025.09 | 0.8787 | — | — | — | |
| TIETesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 0.93 | 0.52 | 54.18 | 123.02 | |
| SSJFTesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 1.47 | 0.71 | 62.93 | 140.8 | |
| LTRTesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 1.76 | 0.83 | 65.33 | 142.17 | |
| FCFSModel=Llama, Workload Mode=Burst Mode2025.09 | 2.2257 | — | — | — | |
| TIETesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 2.81 | 1.54 | 146.14 | 359.08 | |
| FCFSTesting Model Size=8B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 3.62 | 1.45 | 83.65 | 159.72 | |
| SSJFTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 4.32 | 2.06 | 171.14 | 395.78 | |
| LTRTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 5.14 | 2.36 | 174.31 | 394.1 | |
| FCFSTesting Model Size=70B, Request Rate (RPS)=100, Training Dataset=LMSYS-Chat-1M (8B)2026.04 | 11.36 | 4.52 | 235.64 | 444.52 |