LLM Inference Efficiency
Benchmarks
DatasetSOTA methodMetricTrendResultsLast updated
1.05TTFT (s)
24
May 26, 2026
0.37TTFT (s)
24
May 26, 2026
0.2TTFT (s)
24
May 26, 2026
40.6Time To First Token (TTFT)
12
Mar 17, 2026
191.7TTFT (ms)
9
Jun 4, 2026
2Slowdown
8
Jun 11, 2026
2.52Throughput (samples/s)
7
Feb 27, 2026
3.032Prefill Time (sec)
5
Mar 25, 2026
43.36Latency (s)
4
May 19, 2026
160.32Latency (s)
4
May 11, 2026
6.978Average Speedup
2
Jul 7, 2026
5,233.11TTFT (ms)
2
May 27, 2026
162.78Latency (s)
2
May 11, 2026
—
—Throughput (samples/s)
0
Feb 26, 2026