LLM Decoding on Long Context 128K
876.9Throughput (tok/s)SparDA
Evaluation results
| Method | Links | |
|---|---|---|
| SparDAModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 876.9 | |
| SparseModel=NOSA-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 795.9 | |
| SparDAModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 691.3 | |
| SparseModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 585.4 | |
| SparDAModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 526.3 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 501.4 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 485.5 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=64, Offloading=Yes, GPU=NVIDIA A1002026.06 | 447.9 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 424.4 | |
| SparseModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 395.6 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 383.1 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 303.3 | |
| SparDAModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 298.2 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 287.3 | |
| SparseModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 232.4 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 195.1 | |
| SparDAModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 160.2 | |
| SparDAModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 148.4 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 141 | |
| SparseModel=NOSA-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 136.5 | |
| SparseModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 127.4 | |
| SparseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 117.3 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 95.7 | |
| DenseModel=NOSA-8B, Batch size (B)=8, Offloading=No, GPU=NVIDIA A1002026.06 | 75.3 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 70.3 | |
| DenseModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 67.2 | |
| DenseModel=NOSA-8B, Batch size (B)=4, Offloading=No, GPU=NVIDIA A1002026.06 | 66.6 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 62.8 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=8, Offloading=Yes, GPU=NVIDIA A1002026.06 | 62.3 | |
| InfiniGenModel=NOSA-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 62.2 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=16, Offloading=Yes, GPU=NVIDIA A1002026.06 | 58.1 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=4, Offloading=Yes, GPU=NVIDIA A1002026.06 | 49.6 | |
| InfiniGenModel=MiniCPM4.1-8B, Batch size (B)=32, Offloading=Yes, GPU=NVIDIA A1002026.06 | 41.3 |