LLM Serving Latency on Crawler Workload 4.0 QPS, 10x delays
0.64P50 Latency ScorevLLM-NS
Evaluation results
| Method | Links | ||
|---|---|---|---|
| vLLM-NSMode=Non-streaming baseline2026.03 | 0.64 | 8.97 | |
| MCPSEviction Strategy=Swap2026.03 | 3.86 | 0.48 | |
| MCPSEviction Strategy=Cost-Based2026.03 | 4.96 | 0.77 | |
| MCPSEviction Strategy=Recompute2026.03 | 5.92 | 0.73 | |
| vLLM-SEviction Strategy=Swap2026.03 | 7.29 | 0.66 | |
| LCASEviction Strategy=Swap2026.03 | 7.32 | 4.8 | |
| FCFSEviction Strategy=Swap2026.03 | 7.78 | 6.69 | |
| LCASEviction Strategy=Cost-Based2026.03 | 7.82 | 9.14 | |
| vLLM-SEviction Strategy=Cost-Based2026.03 | 8.25 | 0.71 | |
| FCFSEviction Strategy=Cost-Based2026.03 | 8.3 | 8.62 | |
| vLLM-SEviction Strategy=Recompute2026.03 | 8.59 | 0.78 | |
| LCASEviction Strategy=Recompute2026.03 | 8.61 | 9.23 | |
| FCFSEviction Strategy=Recompute2026.03 | 8.77 | 10.03 |