LLM Decoding
Benchmarks
DatasetSOTA methodMetricTrendResultsLast updated
40,000cuBLASLt Throughput
78
Mar 6, 2026
1,218.1Decode Throughput (tok/s)
48
Jun 4, 2026
3,119.55Throughput
48
Feb 26, 2026
1,042.6Decoding Throughput (tok/s)
42
Jun 4, 2026
930.5Decode Throughput (tok/s)
38
Jun 4, 2026
876.9Throughput (tok/s)
33
Jun 4, 2026
894.32Throughput
27
Feb 26, 2026
36.1Throughput (tokens/s)
6
Jun 4, 2026
2.4Latency (ms/token)
5
Mar 13, 2026
0.216Per-step Decoding Latency
4
Feb 26, 2026
0.517Decode Time per Step
4
Feb 26, 2026
0.037EM
3
Feb 26, 2026
0.052E.M.
3
Feb 26, 2026
9.6E.M.
3
Feb 26, 2026
72.8Dense Latency (ms)
1
May 21, 2026
62.6Dense Latency (ms)
1
May 21, 2026
61.2Dense Latency (ms)
1
May 21, 2026
60.9Dense Latency (ms)
1
May 21, 2026