LLM Decoding on Long Context 96K
930.5Decode Throughput (tok/s)SparDA
Evaluation results
| Method | Links | |
|---|---|---|
| SparDAModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 930.5 | |
| SparseModel=NOSA-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 849 | |
| SparseModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 814.2 | |
| SparDAModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 778.4 | |
| SparseModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 629.9 | |
| SparDAModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 568.1 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 529.4 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 522.6 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=128, Offloading=Yes, GPU=NVIDIA A1002026.06 | 503.1 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 467 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 436 | |
| SparseModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 424 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 402.5 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 310.7 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 306.6 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 301.6 | |
| SparDAModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 295.2 | |
| SparseModel=NOSA-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 294.8 | |
| SparseModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 253.5 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 208.7 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 158.7 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 157.8 | |
| SparseModel=NOSA-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 153 | |
| SparDAModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 146.1 | |
| SparseModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 140.7 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 127.6 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 124.7 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 122.6 | |
| DenseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 97.4 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 97 | |
| DenseModel=NOSA-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 96.7 | |
| DenseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 83.3 | |
| DenseModel=NOSA-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 82.9 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 62 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 61.8 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 57.5 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 53.2 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 50.8 |