Fused Matmul and Sampling on Synthetic D=4096, V=151k
1.98Speedup vs Multinomial SamplingFlashSampling
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| FlashSamplingBatch size (B)=128, GPU=B3002026.03 | 1.98 | 2.28 | 1.33 | |
| FlashSamplingBatch size (B)=128, GPU=B2002026.03 | 1.89 | 1.55 | 1.27 | |
| FlashSamplingBatch size (B)=64, GPU=B2002026.03 | 1.84 | 1.67 | 1.39 | |
| FlashSamplingBatch size (B)=64, GPU=B3002026.03 | 1.81 | 2.52 | 1.37 | |
| FlashSamplingBatch size (B)=32, GPU=B2002026.03 | 1.68 | 1.68 | 1.38 | |
| FlashSamplingBatch size (B)=256, GPU=B3002026.03 | 1.65 | 2.04 | 1.09 | |
| FlashSamplingBatch size (B)=32, GPU=B3002026.03 | 1.64 | 2.47 | 1.34 | |
| FlashSamplingBatch size (B)=128, GPU=H1002026.03 | 1.64 | 1.35 | 1.12 | |
| FlashSamplingBatch size (B)=64, GPU=H1002026.03 | 1.6 | 1.46 | 1.26 | |
| FlashSamplingBatch size (B)=64, GPU=H2002026.03 | 1.6 | 1.47 | 1.21 | |
| FlashSamplingBatch size (B)=256, GPU=B2002026.03 | 1.58 | 1.32 | 1.07 | |
| FlashSamplingBatch size (B)=16, GPU=B2002026.03 | 1.57 | 1.65 | 1.36 | |
| FlashSamplingBatch size (B)=16, GPU=B3002026.03 | 1.54 | 2.47 | 1.33 | |
| FlashSamplingBatch size (B)=8, GPU=B2002026.03 | 1.53 | 1.61 | 1.33 | |
| FlashSamplingBatch size (B)=8, GPU=B3002026.03 | 1.5 | 2.46 | 1.3 | |
| FlashSamplingBatch size (B)=4, GPU=B2002026.03 | 1.47 | 1.61 | 1.32 | |
| FlashSamplingBatch size (B)=32, GPU=H2002026.03 | 1.47 | 1.47 | 1.22 | |
| FlashSamplingBatch size (B)=1, GPU=B2002026.03 | 1.46 | 1.51 | 1.32 | |
| FlashSamplingBatch size (B)=1, GPU=B3002026.03 | 1.46 | 2.45 | 1.31 | |
| FlashSamplingBatch size (B)=2, GPU=B2002026.03 | 1.46 | 1.56 | 1.3 | |
| FlashSamplingBatch size (B)=4, GPU=B3002026.03 | 1.44 | 2.37 | 1.3 | |
| FlashSamplingBatch size (B)=2, GPU=B3002026.03 | 1.43 | 2.18 | 1.28 | |
| FlashSamplingBatch size (B)=32, GPU=H1002026.03 | 1.38 | 1.37 | 1.2 | |
| FlashSamplingBatch size (B)=16, GPU=H2002026.03 | 1.37 | 1.45 | 1.21 | |
| FlashSamplingBatch size (B)=128, GPU=H2002026.03 | 1.37 | 1.12 | 0.91 | |
| FlashSamplingBatch size (B)=8, GPU=H2002026.03 | 1.31 | 1.44 | 1.19 | |
| FlashSamplingBatch size (B)=16, GPU=H1002026.03 | 1.3 | 1.35 | 1.18 | |
| FlashSamplingBatch size (B)=256, GPU=H1002026.03 | 1.29 | 1.04 | 0.81 | |
| FlashSamplingBatch size (B)=1, GPU=H2002026.03 | 1.28 | 1.35 | 1.2 | |
| FlashSamplingBatch size (B)=4, GPU=H2002026.03 | 1.27 | 1.41 | 1.18 | |
| FlashSamplingBatch size (B)=8, GPU=H1002026.03 | 1.27 | 1.33 | 1.17 | |
| FlashSamplingBatch size (B)=2, GPU=H2002026.03 | 1.26 | 1.38 | 1.17 | |
| FlashSamplingBatch size (B)=1, GPU=H1002026.03 | 1.25 | 1.3 | 1.18 | |
| FlashSamplingBatch size (B)=2, GPU=H1002026.03 | 1.24 | 1.3 | 1.16 | |
| FlashSamplingBatch size (B)=4, GPU=H1002026.03 | 1.24 | 1.32 | 1.16 | |
| FlashSamplingBatch size (B)=256, GPU=H2002026.03 | 1.17 | 0.98 | 0.78 |