Inference Efficiency
Benchmarks
DatasetSOTA methodMetricTrendResultsLast updated
0.019Relative Overhead (%)
90
Mar 23, 2026
5.33Speedup Factor
90
May 13, 2026
760.74Throughput (tokens/s)
26
Feb 26, 2026
1,067Prefill Throughput (1K) (tokens/s)
20
Feb 26, 2026
-4.27Inference Length
20
Feb 26, 2026
-18.33Sequence Length Delta
20
Feb 26, 2026
15.9Total GEMM Latency (ms)
19
Feb 26, 2026
2.08ATGR
18
Apr 21, 2026
4Storage (MB)
15
Feb 26, 2026
1.889Latency
13
Mar 20, 2026
0.014Relative Execution Time Overhead
12
Mar 23, 2026
0.05Relative Execution Time Overhead
12
Mar 23, 2026
496Time to Last Token (ms)
12
Feb 26, 2026
0.005Inference Latency (s)
12
Feb 26, 2026
499.49Throughput (Samples/s)
11
May 29, 2026
11Inference Time (min)
10
Jul 1, 2026
0.84Prefill TTFT (1k)
10
Mar 18, 2026
159Theoretical Compute (TFLOPs)
10
Mar 12, 2026
528Prefill Latency (ms)
10
Feb 26, 2026
12.46Decode Speed (tokens/sec)
9
Jun 2, 2026
1,480.2Throughput (tokens/s)
9
Mar 17, 2026
4.75#MAT
9
Feb 26, 2026
0.08Encoding Time
8
May 19, 2026
28,298Throughput (tokens/s)
8
May 12, 2026
101TTFT
8
May 8, 2026