LLM Decoding Throughput on NOSA-8B 64K context length
1,996.3Throughput (tok/s)SparDA
Evaluation Results
| Method | Links | |
|---|---|---|
| SparDABatch size=128, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,996.3 | |
| SparDABatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,905.2 | |
| SparseBatch size=128, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,793.1 | |
| SparseBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,553.1 | |
| SparDABatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,392.3 | |
| SparseBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,071.1 | |
| SparDABatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 870.9 | |
| SparseBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 704.6 | |
| SparseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 439.7 | |
| SparDABatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 431.2 | |
| SparseBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 397.9 | |
| SparseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 233.2 | |
| SparseBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 221 | |
| InfiniGenBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 220.4 | |
| SparDABatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 218.8 | |
| DenseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 217.6 | |
| InfiniGenBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 181 | |
| DenseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 174.3 | |
| InfiniGenBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 139.1 | |
| InfiniGenBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 106.2 | |
| InfiniGenBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 78.1 |