LLM Serving on ToolBench
1.11Effective Throughput (req/s)AugServe
Evaluation results
| Method | Links | |
|---|---|---|
| AugServeModel=OPT-13B, GPU=H800, Request rate (req/s)=4.02025.12 | 1.11 | |
| vLLMModel=OPT-13B, GPU=H800, Request rate (req/s)=4.02025.12 | 1.09 | |
| AugServeModel=OPT-13B, GPU=H800, Request rate (req/s)=5.02025.12 | 0.71 | |
| AugServeModel=OPT-13B, GPU=H800, Request rate (req/s)=6.02025.12 | 0.56 | |
| InferCeptModel=OPT-13B, GPU=H800, Request rate (req/s)=4.02025.12 | 0.31 | |
| vLLMModel=OPT-13B, GPU=H800, Request rate (req/s)=5.02025.12 | 0.25 | |
| vLLMModel=OPT-13B, GPU=H800, Request rate (req/s)=6.02025.12 | 0.19 | |
| InferCeptModel=OPT-13B, GPU=H800, Request rate (req/s)=5.02025.12 | 0.16 | |
| InferCeptModel=OPT-13B, GPU=H800, Request rate (req/s)=6.02025.12 | 0.1 |