LLM Inference
Benchmarks
Dataset NameSOTA methodMetricTrendResultsLast Updated
54.3SLO Attainment
9
Feb 26, 2026
119.64Prefill Min Throughput (tokens/sec)
8
May 12, 2026
0.13Latency (ms)
8
Feb 26, 2026
2.61Speedup (Alpaca)
8
Feb 26, 2026
12.8Throughput (RPS)
6
Jun 2, 2026
96.93Throughput (RPS)
6
Jun 2, 2026
58.33Throughput (RPS)
6
Jun 2, 2026
110Throughput (TPS)
6
Apr 21, 2026
185.5Throughput (Tok/s)
6
Apr 6, 2026
46.4Decode Throughput (tok/s)
6
Apr 6, 2026
0.62Prefill Latency (ms)
6
Feb 26, 2026
117.5Throughput (tok/s)
5
Jun 29, 2026
112Throughput (tok/s)
5
Jun 29, 2026
124.7Throughput (tok/s)
5
Jun 29, 2026
384Max Batch Size
5
Feb 26, 2026
0.1Latency (ms)
4
Apr 21, 2026
20Latency (ms)
4
Apr 21, 2026
7.3Throughput (tok/s)
4
Apr 21, 2026
155.3Throughput (tok/s)
4
Apr 6, 2026
51.1Decode Throughput (tok/s)
4
Apr 6, 2026
139.98Latency (ms)
4
Mar 18, 2026
46.71Latency (ms)
4
Mar 18, 2026
2,656.61Throughput (tok/s)
2
Jun 4, 2026
556Memory (MB)
2
May 8, 2026