LLM Inference on ToolBench dataset
100SLO AttainmentAugServe
Evaluation results
| Method | Links | |
|---|---|---|
| AugServeRequest rate=4.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 100 | |
| vLLMRequest rate=4.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 98.4 | |
| AugServeRequest rate=5.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 63.8 | |
| AugServeRequest rate=6.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 50.1 | |
| InferCeptRequest rate=4.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 28.3 | |
| vLLMRequest rate=5.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 22.7 | |
| vLLMRequest rate=6.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 17.8 | |
| InferCeptRequest rate=5.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 14 | |
| InferCeptRequest rate=6.0, Model=OPT-13B, Hardware=H800 GPU2025.12 | 9.3 |