LLM Inference on Merge
1.16Goodput (req/s)AugServe
Evaluation Results
| Method | Links | |
|---|---|---|
| AugServeRequest rate=3, CV=1, Model=OPT-13B, GPU=H8002025.12 | 1.16 | |
| AugServeRequest rate=2, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 1.15 | |
| AugServeRequest rate=2, CV=1, Model=OPT-13B, GPU=H8002025.12 | 0.87 | |
| AugServeRequest rate=3, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 0.58 | |
| AugServeRequest rate=2, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.47 | |
| AugServeRequest rate=3, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.35 | |
| InferCeptRequest rate=2, CV=1, Model=OPT-13B, GPU=H8002025.12 | 0.28 | |
| vLLMRequest rate=2, CV=1, Model=OPT-13B, GPU=H8002025.12 | 0.22 | |
| InferCeptRequest rate=3, CV=1, Model=OPT-13B, GPU=H8002025.12 | 0.2 | |
| vLLMRequest rate=3, CV=1, Model=OPT-13B, GPU=H8002025.12 | 0.15 | |
| InferCeptRequest rate=2, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 0.13 | |
| InferCeptRequest rate=3, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 0.11 | |
| vLLMRequest rate=2, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 0.1 | |
| InferCeptRequest rate=2, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.08 | |
| vLLMRequest rate=3, CV=1.5, Model=OPT-13B, GPU=H8002025.12 | 0.07 | |
| InferCeptRequest rate=3, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.06 | |
| vLLMRequest rate=2, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.05 | |
| vLLMRequest rate=3, CV=2, Model=OPT-13B, GPU=H8002025.12 | 0.04 |