LLM Decoding Throughput on MiniCPM 8B 64K context length 4.1
1,159.1Throughput (tok/s)SparDA
Evaluation Results
| Method | Links | |
|---|---|---|
| SparDABatch size=128, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,159.1 | |
| SparDABatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,049.9 | |
| SparseBatch size=128, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,041.9 | |
| SparDABatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,026.5 | |
| SparseBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 937.2 | |
| SparDABatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 811.8 | |
| SparseBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 756.1 | |
| SparseBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 549.4 | |
| SparDABatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 470.3 | |
| SparseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 466.2 | |
| SparseBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 350.8 | |
| SparseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 246.5 | |
| SparDABatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 234.9 | |
| DenseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 218.3 | |
| SparseBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 213.1 | |
| DenseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 174.1 | |
| InfiniGenBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 166.5 | |
| InfiniGenBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 129.1 | |
| InfiniGenBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 94 | |
| InfiniGenBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 63.3 | |
| InfiniGenBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 55.2 |