LLM Decoding Throughput on MiniCPM 128K context length 4.1-8B
1,000.1Decoding Throughput (tok/s)SparDA
Evaluation Results
| Method | Links | |
|---|---|---|
| SparDABatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,000.1 | |
| SparDABatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 899.2 | |
| SparseBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 788.9 | |
| SparDABatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 705.3 | |
| SparseBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 618.6 | |
| SparDABatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 471.2 | |
| SparseBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 447.9 | |
| SparseBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 279.5 | |
| SparDABatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 240.2 | |
| SparseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 189.5 | |
| SparseBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 167.8 | |
| InfiniGenBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 117.5 | |
| DenseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 108.6 | |
| InfiniGenBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 85.6 | |
| InfiniGenBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 66.5 | |
| InfiniGenBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 51.8 |