LLM Serving Efficiency on LMSYS trace
246GPUs UsedToken-budget
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Token-budgetRequest Rate=1,000 req/s, B_short=81922026.04 | 246 | 31.3 | 1.48 | |
| HomogeneousRequest Rate=1,000 req/s, B_short=81922026.04 | 358 | — | 1.45 |
Loading the SOTA2 catalog…