LLM Decoding on Long Context 64K
1,042.6Decoding Throughput (tok/s)SparDA
Evaluation Results
| Method | Links | |
|---|---|---|
| SparDAModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 1,042.6 | |
| SparDAModel=NOSA-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 961.3 | |
| SparseModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 911.8 | |
| SparDAModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 887.8 | |
| SparseModel=NOSA-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 861.3 | |
| SparseModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 704.8 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 593.1 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 578.4 | |
| SparDAModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 577.3 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 573.7 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 556.6 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 516.1 | |
| SparseModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 479.8 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 457.8 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 443.9 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 345.6 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 344.6 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 316.3 | |
| SparseModel=NOSA-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 310.3 | |
| SparDAModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 292.8 | |
| SparseModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 284.6 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 236.7 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 173.2 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 160.3 | |
| SparseModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 158.5 | |
| SparseModel=NOSA-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 157.4 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 147.3 | |
| SparDAModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 146.9 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 138.1 | |
| DenseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 134.8 | |
| DenseModel=NOSA-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 133.7 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 133.5 | |
| DenseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 109.6 | |
| DenseModel=NOSA-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 109.5 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 100.1 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 67.5 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 66.9 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 63.3 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 63.3 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 62.5 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 54 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 27.2 |