Large Language Model Inference
Benchmarks
DatasetSOTA methodMetricTrendResultsLast updated
8.23P99 Per-Token Latency (ms)
33
Apr 28, 2026
104TPS
20
Feb 26, 2026
0.152Latency (s)
18
Apr 7, 2026
1,779Throughput
18
Mar 20, 2026
2,341.3Throughput
18
Mar 20, 2026
0.025Prefill Time (s)
15
Apr 7, 2026
265.7Throughput (TPS)
14
Feb 26, 2026
0.05TTFT
8
Jun 17, 2026
37.29Throughput
7
May 19, 2026
81.35Q Score
6
May 20, 2026
0.255Storage (GB)
6
May 7, 2026
130.7Throughput (Tok/s)
5
Jun 11, 2026
1.94TPOTH
4
Mar 17, 2026
9Throughput (Intel i7-13620H)
3
Jun 23, 2026
17.02Throughput (req/s)
2
Apr 23, 2026
48.1Latency (ms)
2
Apr 14, 2026