Language Model Decoding Efficiency on GSM8K, TriviaQA, MBPP, and MMLU Efficiency Subset
74.1Accuracy (GSM8K)SimSD
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| SimSDBL (Block Length)=8, CUDA Graph (CG)=true2026.06 | 74.1 | 88.3 | 80.4 | 84.4 | 81.8 | 5.4 | |
| SimSDBL (Block Length)=4, CUDA Graph (CG)=true2026.06 | 72.6 | 73.3 | 72.2 | 68.4 | 71.6 | 7.46 | |
| S2D2BL (Block Length)=8, CUDA Graph (CG)=true2026.06 | 57.6 | 30 | 50.2 | 28 | 41.5 | 2.74 | |
| S2D2BL (Block Length)=4, CUDA Graph (CG)=true2026.06 | 54.8 | 28.3 | 47.7 | 32.3 | 40.8 | 4.25 | |
| Vanilla + CGBL (Block Length)=4, CUDA Graph (CG)=true, Tensor Parallelism (TP)=size 22026.06 | 50.1 | 50.5 | 50.7 | 56.04 | 51.8 | 5.4 | |
| Vanilla + CGBL (Block Length)=8, CUDA Graph (CG)=true, Tensor Parallelism (TP)=size 22026.06 | 30.9 | 31.9 | 30.4 | 36.8 | 32.5 | 2.14 | |
| VanillaBL (Block Length)=8, CUDA Graph (CG)=false, Tensor Parallelism (TP)=size 22026.06 | 15.3 | 20.4 | 18.7 | 6.21 | 15.2 | 1 | |
| VanillaBL (Block Length)=4, CUDA Graph (CG)=false, Tensor Parallelism (TP)=size 22026.06 | 10 | 11.4 | 11.4 | 5.59 | 9.6 | 1 |