LLM Decoding Throughput on MiniCPM 8B 96K context length 4.1
1,076.3Decoding Throughput (tok/s)SparDA
Evaluation Results
| Method | Links | |
|---|---|---|
| SparDABatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,076.3 | |
| SparDABatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 939.2 | |
| SparseBatch size=128, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 934.3 | |
| SparseBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 839.5 | |
| SparDABatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 752.9 | |
| SparseBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 679.4 | |
| SparseBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 489 | |
| SparDABatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 475.6 | |
| SparseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 401.4 | |
| SparseBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 306 | |
| SparDABatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 238.6 | |
| SparseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 215.5 | |
| SparseBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 185.7 | |
| DenseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 159 | |
| DenseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 133.8 | |
| InfiniGenBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 117.9 | |
| InfiniGenBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 86 | |
| InfiniGenBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 64.9 | |
| InfiniGenBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 49.4 |