Online Inference on ShareGPT
27P50 LatencySGLang-Non-Det
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| SGLang-Non-DetQPS=122026.01 | 27 | 35.4 | 50.3 | — | — | — | — | — | |
| LLM-42QPS=12, Workload Determinism=2%2026.01 | 27.4 | 35.8 | 52.5 | — | — | — | — | — | |
| LLM-42QPS=12, Workload Determinism=5%2026.01 | 27.6 | 36.1 | 54.1 | — | — | — | — | — | |
| LLM-42QPS=12, Workload Determinism=10%2026.01 | 27.9 | 37.1 | 54 | — | — | — | — | — | |
| SGLang-Non-DetQPS=142026.01 | 27.9 | 37 | 55.8 | — | — | — | — | — | |
| LLM-42QPS=14, Workload Determinism=2%2026.01 | 27.9 | 37.5 | 57 | — | — | — | — | — | |
| LLM-42QPS=14, Workload Determinism=5%2026.01 | 28.2 | 37.8 | 56.5 | — | — | — | — | — | |
| SGLang-Non-DetQPS=162026.01 | 28.8 | 39 | 60.5 | — | — | — | — | — | |
| LLM-42QPS=16, Workload Determinism=2%2026.01 | 28.8 | 39.7 | 61.7 | — | — | — | — | — | |
| LLM-42QPS=14, Workload Determinism=10%2026.01 | 28.9 | 39.5 | 58.6 | — | — | — | — | — | |
| LLM-42QPS=16, Workload Determinism=5%2026.01 | 29.2 | 40.5 | 60.9 | — | — | — | — | — | |
| LLM-42QPS=12, Workload Determinism=20%2026.01 | 29.3 | 41.1 | 57.8 | — | — | — | — | — | |
| LLM-42QPS=16, Workload Determinism=10%2026.01 | 29.8 | 41.6 | 63.3 | — | — | — | — | — | |
| SGLang-Non-DetQPS=182026.01 | 29.8 | 41.4 | 65.2 | — | — | — | — | — | |
| LLM-42QPS=18, Workload Determinism=2%2026.01 | 30.2 | 42.7 | 65.9 | — | — | — | — | — | |
| LLM-42QPS=18, Workload Determinism=5%2026.01 | 30.6 | 43.3 | 67.6 | — | — | — | — | — | |
| LLM-42QPS=14, Workload Determinism=20%2026.01 | 31.1 | 45.1 | 63.8 | — | — | — | — | — | |
| LLM-42QPS=18, Workload Determinism=10%2026.01 | 32.2 | 46.3 | 70.2 | — | — | — | — | — | |
| LLM-42QPS=16, Workload Determinism=20%2026.01 | 35.2 | 51.2 | — | — | — | — | — | — | |
| LLM-42QPS=12, Workload Determinism=50%2026.01 | 38.5 | 52 | 68.1 | — | — | — | — | — | |
| LLM-42QPS=18, Workload Determinism=20%2026.01 | 40.7 | 57.5 | 79.3 | — | — | — | — | — | |
| LLM-42QPS=14, Workload Determinism=50%2026.01 | 42.5 | 57 | 75.8 | — | — | — | — | — | |
| LLM-42QPS=12, Workload Determinism=100%2026.01 | 45.4 | 58 | 73.9 | — | — | — | — | — | |
| LLM-42QPS=16, Workload Determinism=50%2026.01 | 46.1 | 61 | 83.6 | — | — | — | — | — | |
| LLM-42QPS=14, Workload Determinism=100%2026.01 | 48.2 | 61.6 | 82 | — | — | — | — | — | |
| LLM-42QPS=18, Workload Determinism=50%2026.01 | 50.7 | 67.3 | 90.5 | — | — | — | — | — | |
| LLM-42QPS=16, Workload Determinism=100%2026.01 | 51.2 | 65.9 | 87.8 | — | — | — | — | — | |
| SGLang-DetQPS=122026.01 | 53.4 | 69.8 | 107.8 | — | — | — | — | — | |
| LLM-42QPS=18, Workload Determinism=100%2026.01 | 57.1 | 75.4 | 101.2 | — | — | — | — | — | |
| SGLang-DetQPS=142026.01 | 60.1 | 79.9 | 127 | — | — | — | — | — | |
| SGLang-DetQPS=162026.01 | 62.5 | 86 | 138.6 | — | — | — | — | — | |
| SGLang-DetQPS=182026.01 | 76.2 | 105.6 | 171.6 | — | — | — | — | — | |
| AXLearnModel=Llama 2 7B, Hardware=Google Cloud TPU v5p-8 VM, Max Input Length=1024, Max Output Length=2562025.07 | — | — | — | 40.1 | 9.1 | — | — | — | |
| AXLearnModel=Llama 2 70B, Hardware=Google Cloud TPU v6e-8 VM, Max Input Length=1800, Max Output Length=2562025.07 | — | — | — | 150.5 | 28.1 | — | — | — | |
| BaselineHardware=Jetson Orin NX 16GB, Backend=llama.cpp, Number of parallel requests (-np)=1, Sequence length (-n)=128, Memory mapping (--mmap)=true2026.05 | — | — | — | 7,150.12 | 554.69 | — | — | — | |
| BaselineHardware=RTX 3090, moe-expert-cache-size=6, Framework=vLLM2026.05 | — | — | — | 769.23 | 254.31 | 3.58 | — | 39.4 | |
| CE-onlyHardware=RTX 3090, moe-expert-cache-size=6, Framework=vLLM2026.05 | — | — | — | 780.12 | 286.82 | 2.95 | — | 21.1 | |
| ReMoEHardware=Jetson Orin NX 16GB, Backend=llama.cpp, Number of parallel requests (-np)=1, Sequence length (-n)=128, Memory mapping (--mmap)=true2026.05 | — | — | — | 5,368.77 | 306.27 | — | 1.81 | — | |
| ReMoEHardware=RTX 3090, moe-expert-cache-size=6, Framework=vLLM2026.05 | — | — | — | 758.27 | 242.99 | 3.88 | — | 43.4 | |
| vLLMModel=Llama 2 7B, Hardware=Google Cloud TPU v5p-8 VM, Max Input Length=1024, Max Output Length=2562025.07 | — | — | — | 538.6 | 22.4 | — | — | — | |
| vLLMModel=Llama 2 70B, Hardware=Google Cloud TPU v6e-8 VM, Max Input Length=1800, Max Output Length=2562025.07 | — | — | — | 80 | 189.8 | — | — | — |