LLM Inference Performance
Benchmarks
DatasetSOTA methodMetricTrendResultsLast updated
18.7TPOT Speedup (%)
28
Mar 18, 2026
56.03TTFT (ms)
12
Mar 10, 2026
256Max Concurrency
8
Jun 5, 2026
10.7Prefill Time (s)
3
Mar 13, 2026
5.66Prefill Time (s)
3
Mar 13, 2026
2.59Prefill Time (s)
3
Mar 13, 2026
0.45Prefill Time (s)
3
Mar 13, 2026