LLM Serving Latency on ANNS Workload 2.0 QPS, 30x delays
0.72P50 LatencyvLLM-NS
Evaluation results
| Method | Links | ||
|---|---|---|---|
| vLLM-NSMode=Non-streaming baseline2026.03 | 0.72 | 6.56 | |
| MCPSEviction Strategy=Swap2026.03 | 1.47 | 0.22 | |
| LCASEviction Strategy=Swap2026.03 | 1.62 | 0.97 | |
| MCPSEviction Strategy=Recompute2026.03 | 2.2 | 0.19 | |
| MCPSEviction Strategy=Cost-Based2026.03 | 2.23 | 0.19 | |
| vLLM-SEviction Strategy=Swap2026.03 | 2.26 | 0.18 | |
| FCFSEviction Strategy=Swap2026.03 | 2.37 | 1.26 | |
| LCASEviction Strategy=Recompute2026.03 | 2.38 | 1.79 | |
| LCASEviction Strategy=Cost-Based2026.03 | 2.44 | 1.79 | |
| vLLM-SEviction Strategy=Recompute2026.03 | 2.53 | 0.19 | |
| vLLM-SEviction Strategy=Cost-Based2026.03 | 2.63 | 0.19 | |
| FCFSEviction Strategy=Recompute2026.03 | 2.7 | 1.84 | |
| FCFSEviction Strategy=Cost-Based2026.03 | 2.7 | 2.04 |