LLM Serving
Benchmarks
DatasetSOTA methodMetricTrendResultsLast updated
443TTFT (ms)
27
Apr 21, 2026
878Time To First Token (ms)
27
Apr 21, 2026
154.52Average Latency (ms/token)
12
Jun 29, 2026
107.1Average Latency (ms/token)
12
Jun 29, 2026
111.46Average Latency (ms/token)
12
Jun 29, 2026
234.09p90 Latency (ms/token)
12
Jun 29, 2026
157.83p90 Latency (ms/token)
12
Jun 29, 2026
21.39TTFT P50 (s)
12
May 12, 2026
1.11Effective Throughput (req/s)
9
Feb 26, 2026
0.6Effective Throughput (req/s)
9
Feb 26, 2026
12.8Throughput (req/s)
6
Jun 2, 2026
45.2TTFT (ms)
4
Feb 26, 2026
91.13SLO Attainment
3
Jun 30, 2026
8.02Throughput (req/s)
2
Jun 2, 2026
26.3Efficiency Gain
1
May 21, 2026