Snippet Generation on Industry Workload Llama-3.1-8B
8.67Throughput (A100)BatchLLM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| BatchLLM2024.11 | 8.67 | 4.24 | |
| vLLM + c + pchunked-prefill=true, prefix-caching=true2024.11 | 6.71 | 3.36 | |
| vLLM + pprefix-caching=true2024.11 | 6.2 | 3.27 | |
| vLLM + cchunked-prefill=true2024.11 | 5.48 | 2.84 | |
| vLLM2024.11 | 5.45 | 2.75 |