LLM Decoding Throughput on NOSA-8B 96K context length
1,694.1Throughput (tok/s)SparDA
Evaluation Results
| Method | Links | |
|---|---|---|
| SparDABatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,694.1 | |
| SparseBatch size=128, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,580.5 | |
| SparseBatch size=64, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,354.3 | |
| SparDABatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 1,263.7 | |
| SparseBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 962.4 | |
| SparDABatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 852.9 | |
| SparseBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 605.5 | |
| SparDABatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 437.6 | |
| SparseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 380.1 | |
| SparseBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 341.3 | |
| SparDABatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 215.5 | |
| SparseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 204.1 | |
| SparseBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 191.5 | |
| InfiniGenBatch size=32, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 165.2 | |
| DenseBatch size=8, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 158.3 | |
| DenseBatch size=4, Offloading=Disabled, Hardware=NVIDIA H1002026.06 | 130 | |
| InfiniGenBatch size=16, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 129.1 | |
| InfiniGenBatch size=8, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 104 | |
| InfiniGenBatch size=4, Offloading=Enabled, Hardware=NVIDIA H1002026.06 | 75.1 |