LLM Serving on τ-bench (trace)
21.39TTFT P50 (s)ContextPilot
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ContextPilotQPS=30, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 21.39 | 39.76 | 42.5 | 58.88 | 74.2 | 67.9 | |
| PRISMQPS=30, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 21.41 | 36.41 | 38.95 | 41.27 | 76.4 | 75.3 | |
| k-LPMQPS=30, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 21.6 | 41.79 | 43.9 | 54.96 | 74 | 67.7 | |
| k-LPMQPS=40, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 30.27 | 55.76 | 59.73 | 94.09 | 74.1 | 67.8 | |
| ContextPilotQPS=40, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 30.28 | 55.42 | 59.56 | 65.7 | 74.1 | 67.8 | |
| PRISMQPS=40, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 30.4 | 52.4 | 55.1 | 57.86 | 76.4 | 75.1 | |
| PRISMQPS=50, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 35.42 | 62.07 | 65.36 | 68.68 | 76.4 | 75.1 | |
| k-LPMQPS=50, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 36.36 | 64.64 | 69.12 | 91.01 | 74.2 | 67.8 | |
| ContextPilotQPS=50, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 36.98 | 65.19 | 69.68 | 75.07 | 74.1 | 67.6 | |
| native SGLangQPS=30, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 57.53 | 105.06 | 111.37 | 131.66 | 73 | 64.2 | |
| native SGLangQPS=40, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 65.53 | 120.19 | 127.47 | 140.71 | 73.1 | 64.5 | |
| native SGLangQPS=50, Model=Qwen3-4B-Instruct-2507, Decode Budget=32, Max Prefill Length=40,9602026.05 | 72.05 | 130.46 | 138.21 | 150.86 | 73.1 | 64.6 |