LLM Decoding Throughput on MiniCPM4.1-8B (32K context)
1,358.2Throughput (tok/s)SparDA
Evaluation Results
| Method | Links | |
|---|---|---|
| SparDABatch size=128, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,358.2 | |
| SparDABatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,328 | |
| SparseBatch size=128, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,225.1 | |
| SparDABatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,205.2 | |
| SparseBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,088.4 | |
| SparseBatch size=16, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 1,073 | |
| SparDABatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 918.5 | |
| SparseBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 861.8 | |
| SparseBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 661.5 | |
| SparseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 550.9 | |
| SparDABatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 471.9 | |
| DenseBatch size=16, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 440.2 | |
| SparseBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 420.9 | |
| DenseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 352.2 | |
| SparseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 277.2 | |
| DenseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 248.3 | |
| SparDABatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 235.7 | |
| SparseBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 224.4 | |
| InfiniGenBatch size=128, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 222.4 | |
| InfiniGenBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 184.8 | |
| InfiniGenBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 140.4 | |
| InfiniGenBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 98.8 | |
| InfiniGenBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 75.1 | |
| InfiniGenBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 56.3 |