ResearchBenchmarksLarge Language Model Inference on Synthetic heavy-tail workload Pareto distributionLarge Language Model Inference on Synthetic heavy-tail workload Pareto distributionFollow17.02Throughput (req/s)BatchLLM4.78967.964811.1414.3152Nov 29, 2024Evaluation resultsMethodMethodLinksThroughput (req/s)SpeedupBatchLLMModel=Qwen 2.5 7B, Pre...Model=Qwen 2.5 7B, Precision=FP16, Input length=2200 tokens, Max batched tokens=2048, KV cache block size=642024.1117.023.2vLLMModel=Qwen 2.5 7B, Pre...Model=Qwen 2.5 7B, Precision=FP16, Input length=2200 tokens, Max batched tokens=2048, KV cache block size=642024.115.261