LLM Serving Efficiency on Azure trace
358GPUs UsedHomogeneous
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| HomogeneousRequest Rate=1,000 req/s, B_short=81922026.04 | 358 | — | 1.82 | |
| Token-budgetRequest Rate=1,000 req/s, B_short=81922026.04 | 208 | 41.9 | 1.71 |
Loading the SOTA2 catalog…