LLM Decoding on Long Context 32K
1,218.1Decode Throughput (tok/s)SparDA
Evaluation results
| Method | Links | |
|---|---|---|
| SparDAModel=NOSA-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 1,218.1 | |
| SparDAModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 1,211.1 | |
| SparseModel=NOSA-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 1,113.3 | |
| SparseModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 1,097.9 | |
| SparDAModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 988.6 | |
| SparseModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 828.2 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 691.9 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=No, GPU=NVIDIA A1002026.06 | 688.7 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 668.9 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 649.2 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 649.2 | |
| SparseModel=NOSA-8B, Batch size (B)=16, Offloading=No, GPU=NVIDIA A1002026.06 | 630.3 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 597.9 | |
| SparDAModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 591.4 | |
| SparseModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 572.7 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 568 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 515.7 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 405.9 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 348.6 | |
| SparseModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 317.8 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 316.4 | |
| SparseModel=NOSA-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 307.8 | |
| SparDAModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 292.4 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 276.1 | |
| DenseModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=No, GPU=NVIDIA A1002026.06 | 268.8 | |
| DenseModel=NOSA-8B, Batch size (B)=16, Offloading=No, GPU=NVIDIA A1002026.06 | 266.8 | |
| DenseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 219.7 | |
| DenseModel=NOSA-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 218.6 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 182.3 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 171.4 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 165.6 | |
| DenseModel=NOSA-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 161 | |
| DenseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 160.9 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 160.4 | |
| SparseModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 158.5 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 157.8 | |
| SparseModel=NOSA-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 155.7 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 154.4 | |
| SparDAModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 148.1 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 119.8 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 108.1 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 75.2 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 72.3 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 71.8 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 71.3 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 55.2 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 46.8 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 27.3 |