LLM Inference on Merge dataset
54.3SLO AttainmentAugServe
Evaluation results
| Method | Links | |
|---|---|---|
| AugServeRequest rate=2.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 54.3 | |
| AugServeRequest rate=3.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 51.3 | |
| AugServeRequest rate=4.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 45.8 | |
| InferCeptRequest rate=2.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 28.3 | |
| vLLMRequest rate=2.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 19.5 | |
| InferCeptRequest rate=3.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 19.1 | |
| InferCeptRequest rate=4.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 14.2 | |
| vLLMRequest rate=3.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 13.8 | |
| vLLMRequest rate=4.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 9.7 |