LLM Serving on Merge dataset
0.6Effective Throughput (req/s)AugServe
Evaluation results
| Method | Links | |
|---|---|---|
| AugServeModel=OPT-13B, GPU=H800, Request rate (req/s)=2.02025.12 | 0.6 | |
| AugServeModel=OPT-13B, GPU=H800, Request rate (req/s)=3.02025.12 | 0.57 | |
| AugServeModel=OPT-13B, GPU=H800, Request rate (req/s)=4.02025.12 | 0.51 | |
| InferCeptModel=OPT-13B, GPU=H800, Request rate (req/s)=2.02025.12 | 0.27 | |
| vLLMModel=OPT-13B, GPU=H800, Request rate (req/s)=2.02025.12 | 0.22 | |
| InferCeptModel=OPT-13B, GPU=H800, Request rate (req/s)=3.02025.12 | 0.21 | |
| InferCeptModel=OPT-13B, GPU=H800, Request rate (req/s)=4.02025.12 | 0.16 | |
| vLLMModel=OPT-13B, GPU=H800, Request rate (req/s)=3.02025.12 | 0.15 | |
| vLLMModel=OPT-13B, GPU=H800, Request rate (req/s)=4.02025.12 | 0.11 |