Decode Latency Benchmark
6.2Decode Latency (ms/tok)Dense Softmax 1.3B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Dense Softmax 1.3BContext=256, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 6.2 | — | |
| Dense Softmax 1.3BContext=1,024, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 6.9 | 12 | |
| Dense Softmax 1.3BContext=4,096, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 9.6 | 55 | |
| Dense Softmax 1.3BContext=8,192, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 12.9 | 108 | |
| Dense Softmax 1.3BContext=16,384, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 19.6 | 216 | |
| Dense Softmax 1.3BContext=24,576, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 26.5 | 328 | |
| MoL Hybrid 2.08BContext=8,192, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 62.5 | — | |
| MoL Hybrid 2.08BContext=256, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 63.2 | — | |
| MoL Hybrid 2.08BContext=4,096, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 63.3 | — | |
| MoL Hybrid 2.08BContext=1,024, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 64.2 | — | |
| MoL Hybrid 2.08BContext=24,576, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 65 | — | |
| MoL Hybrid 2.08BContext=16,384, Batch Size=1, Hardware=RTX 3090, KV Cache Status=Enabled2026.05 | 65.2 | — |