Inference Throughput
Benchmarks
DatasetSOTA methodMetricTrendResultsLast updated
120.62Throughput (k tokens/sec)
30
Mar 23, 2026
80.63Throughput (k tokens/sec)
30
Mar 23, 2026
6,581.55Prefill (tokens/s)
18
Jun 9, 2026
1,484Inference Throughput (tokens/s)
16
May 15, 2026
8,191.3Prefill Throughput (tok/s)
10
Jun 4, 2026
26.74Throughput (tokens/s)
9
Feb 26, 2026
6,991Throughput (TPS)
8
Mar 19, 2026
57,117.4Original Throughput (tokens/s)
6
May 15, 2026
5,888Throughput (token/s)
6
Feb 26, 2026
3,348.18Prefill Throughput (tokens/s)
5
Jun 9, 2026
13,890Throughput (2k Input)
5
Mar 4, 2026
1,935Throughput (Seq Len 256)
4
Jul 7, 2026
1,020Decode Throughput (tok/s)
4
Mar 31, 2026
113.4Throughput (TOPS)
4
Feb 26, 2026
115.2Throughput (Tokens/s)
4
Feb 26, 2026
215.6Throughput (TOK/s)
4
Feb 26, 2026
310.5Throughput (Tokens/sec)
4
Feb 26, 2026
224Max Batch Size
3
May 11, 2026