List your company
ResearchFormerly WizWand

LLM Inference on ToolBench (Goodput)

3.9Goodput (req/s)AugServe
-0.1040.93551.9753.0145Dec 3, 2025
Updated 8mo ago

Evaluation results

MethodLinks
AugServeRequest rate=4, CV=1, Model=OPT-13B, GPU=H8002025.123.9
AugServeRequest rate=3, CV=1, Model=OPT-13B, GPU=H8002025.122.92
vLLMRequest rate=3, CV=1, Model=OPT-13B, GPU=H8002025.122.91
vLLMRequest rate=4, CV=1, Model=OPT-13B, GPU=H8002025.122.81
InferCeptRequest rate=3, CV=1, Model=OPT-13B, GPU=H8002025.122.8
AugServeRequest rate=3, CV=1.5, Model=OPT-13B, GPU=H8002025.122.04
AugServeRequest rate=4, CV=1.5, Model=OPT-13B, GPU=H8002025.121.65
AugServeRequest rate=4, CV=2, Model=OPT-13B, GPU=H8002025.121.51
AugServeRequest rate=3, CV=2, Model=OPT-13B, GPU=H8002025.121.39
InferCeptRequest rate=4, CV=1, Model=OPT-13B, GPU=H8002025.121.06
vLLMRequest rate=3, CV=1.5, Model=OPT-13B, GPU=H8002025.120.16
vLLMRequest rate=4, CV=1.5, Model=OPT-13B, GPU=H8002025.120.14
vLLMRequest rate=3, CV=2, Model=OPT-13B, GPU=H8002025.120.11
InferCeptRequest rate=3, CV=1.5, Model=OPT-13B, GPU=H8002025.120.1
vLLMRequest rate=4, CV=2, Model=OPT-13B, GPU=H8002025.120.08
InferCeptRequest rate=4, CV=1.5, Model=OPT-13B, GPU=H8002025.120.07
InferCeptRequest rate=3, CV=2, Model=OPT-13B, GPU=H8002025.120.05
InferCeptRequest rate=4, CV=2, Model=OPT-13B, GPU=H8002025.120.05