LLM Serving on Synthetic Workloads (Maximum Serving Capacity)
12.8Throughput (req/s)DuetServe
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| DuetServeISL=4096, OSL=64, ISL/OSL=642025.11 | 12.8 | 105 | 1.28 | |
| vLLMISL=4096, OSL=64, ISL/OSL=642025.11 | 10 | 170 | — | |
| DuetServeISL=4096, OSL=1024, ISL/OSL=42025.11 | 9.8 | 50 | 1.11 | |
| vLLMISL=4096, OSL=1024, ISL/OSL=42025.11 | 8.8 | 55 | — | |
| DuetServeISL=4096, OSL=2048, ISL/OSL=22025.11 | 7.2 | 44 | 1.04 | |
| vLLMISL=4096, OSL=2048, ISL/OSL=22025.11 | 6.9 | 45 | — |