LLM Inference on ToolBench (Goodput)
3.9Goodput (req/s)AugServe
Evaluation Results
| Method | Links | |
|---|---|---|
| AugServeRequest rate=4, CV=1, Model=OPT-13B, GPU=H8002025.12 | 3.9 | |
| AugServeRequest rate=3, CV=1, Model=OPT-13B, GPU=H8002025.12 | 2.92 | |
| vLLMRequest rate=3, CV=1, Model=OPT-13B, GPU=H8002025.12 | 2.91 | |
| vLLMRequest rate=4, CV=1, Model=OPT-13B, GPU=H8002025.12 | 2.81 | |
| InferCeptRequest rate=3, CV=1, Model=OPT-13B, GPU=H8002025.12 | 2.8 | |
| AugServeRequest rate=3, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 2.04 | |
| AugServeRequest rate=4, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 1.65 | |
| AugServeRequest rate=4, CV=2, Model=OPT-13B, GPU=H8002025.12 | 1.51 | |
| AugServeRequest rate=3, CV=2, Model=OPT-13B, GPU=H8002025.12 | 1.39 | |
| InferCeptRequest rate=4, CV=1, Model=OPT-13B, GPU=H8002025.12 | 1.06 | |
| vLLMRequest rate=3, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 0.16 | |
| vLLMRequest rate=4, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 0.14 | |
| vLLMRequest rate=3, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.11 | |
| InferCeptRequest rate=3, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 0.1 | |
| vLLMRequest rate=4, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.08 | |
| InferCeptRequest rate=4, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 0.07 | |
| InferCeptRequest rate=3, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.05 | |
| InferCeptRequest rate=4, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.05 |