LLM Decoding Throughput on NOSA-8B 128K context length
1,463.3Throughput (tok/s)SparDA
Evaluation Results
| Method | Links | |
|---|---|---|
| SparDABatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,463.3 | |
| SparseBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,298.3 | |
| SparDABatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,127 | |
| SparseBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 898.7 | |
| SparDABatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 735 | |
| SparseBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 529.2 | |
| SparDABatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 399.3 | |
| SparseBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 285.4 | |
| SparDABatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 219 | |
| SparseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 179.3 | |
| SparseBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 173.2 | |
| InfiniGenBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 166.9 | |
| InfiniGenBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 131.7 | |
| DenseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 108.4 | |
| InfiniGenBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 105.2 | |
| InfiniGenBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 77.5 |