Large Language Model Throughput on ShareGPT v3
8.85Throughput (req/s)vLLM-RA
Evaluation results
| Method | Links | |
|---|---|---|
| vLLM-RAModel=Phi-2 (2.7B), System prompt length=5122024.02 | 8.85 | |
| vLLM-RAModel=Phi-2 (2.7B), System prompt length=10242024.02 | 8.76 | |
| vLLM-PSModel=Phi-2 (2.7B), System prompt length=5122024.02 | 6.29 | |
| vLLM-RAModel=Mistral-7B, System prompt length=5122024.02 | 5.9 | |
| vLLM-RAModel=Mistral-7B, System prompt length=10242024.02 | 5.85 | |
| vLLM-PSModel=Mistral-7B, System prompt length=5122024.02 | 5.4 | |
| vLLMModel=Phi-2 (2.7B), System prompt length=5122024.02 | 5.03 | |
| vLLM-PSModel=Mistral-7B, System prompt length=10242024.02 | 4.92 | |
| vLLM-PSModel=Phi-2 (2.7B), System prompt length=10242024.02 | 4.82 | |
| vLLMModel=Mistral-7B, System prompt length=5122024.02 | 3.68 | |
| vLLM-RAModel=Llama-30B, System prompt length=512, Hardware configuration=Two A100-SXM4-80GB GPUs2024.02 | 3.65 | |
| vLLM-RAModel=Llama-30B, System prompt length=1024, Hardware configuration=Two A100-SXM4-80GB GPUs2024.02 | 3.64 | |
| vLLMModel=Phi-2 (2.7B), System prompt length=10242024.02 | 3.54 | |
| vLLM-PSModel=Llama-30B, System prompt length=512, Hardware configuration=Two A100-SXM4-80GB GPUs2024.02 | 3.01 | |
| vLLMModel=Mistral-7B, System prompt length=10242024.02 | 2.6 | |
| vLLM-PSModel=Llama-30B, System prompt length=1024, Hardware configuration=Two A100-SXM4-80GB GPUs2024.02 | 2.55 | |
| vLLMModel=Llama-30B, System prompt length=512, Hardware configuration=Two A100-SXM4-80GB GPUs2024.02 | 2.15 | |
| vLLM-RAModel=Llama2-13B, System prompt length=5122024.02 | 1.71 | |
| vLLM-RAModel=Llama2-13B, System prompt length=10242024.02 | 1.69 | |
| vLLMModel=Llama-30B, System prompt length=1024, Hardware configuration=Two A100-SXM4-80GB GPUs2024.02 | 1.52 | |
| vLLM-PSModel=Llama2-13B, System prompt length=5122024.02 | 1.44 | |
| vLLM-PSModel=Llama2-13B, System prompt length=10242024.02 | 1.23 | |
| vLLMModel=Llama2-13B, System prompt length=5122024.02 | 0.99 | |
| vLLMModel=Llama2-13B, System prompt length=10242024.02 | 0.66 |