Kernel Speedup on Synthetic Large Configuration (D=8192, V=128k)
1.88SpeedupFlashSampling
Evaluation results
| Method | Links | |
|---|---|---|
| FlashSamplingBatch Size (B)=64, GPU Architecture=B300, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.88 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=B300, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.79 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=B300, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.77 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=B300, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.77 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=B300, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.75 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=B300, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.72 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=B300, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.64 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=B200, Baseline Method=Multinomial Sampling2026.03 | 1.54 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=B300, Baseline Method=Multinomial Sampling2026.03 | 1.52 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=B300, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.51 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=B300, Baseline Method=Multinomial Sampling2026.03 | 1.49 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=B200, Baseline Method=Multinomial Sampling2026.03 | 1.44 | |
| FlashSamplingBatch Size (B)=256, GPU Architecture=B300, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.44 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=B200, Baseline Method=Multinomial Sampling2026.03 | 1.43 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=B200, Baseline Method=Multinomial Sampling2026.03 | 1.42 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=B200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.42 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=B300, Baseline Method=Multinomial Sampling2026.03 | 1.39 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=B300, Baseline Method=Multinomial Sampling2026.03 | 1.39 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=H200, Baseline Method=Multinomial Sampling2026.03 | 1.39 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=B200, Baseline Method=Multinomial Sampling2026.03 | 1.38 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=H100, Baseline Method=Multinomial Sampling2026.03 | 1.36 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=B300, Baseline Method=Multinomial Sampling2026.03 | 1.35 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=B200, Baseline Method=Multinomial Sampling2026.03 | 1.35 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=B200, Baseline Method=Multinomial Sampling2026.03 | 1.34 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=B300, Baseline Method=Multinomial Sampling2026.03 | 1.34 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=B200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.34 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=B300, Baseline Method=Multinomial Sampling2026.03 | 1.33 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=B200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.32 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=B200, Baseline Method=Multinomial Sampling2026.03 | 1.32 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=B200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.31 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=H200, Baseline Method=Multinomial Sampling2026.03 | 1.31 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=B300, Baseline Method=Multinomial Sampling2026.03 | 1.3 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=B200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.3 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=H200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.3 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=H100, Baseline Method=Multinomial Sampling2026.03 | 1.29 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=H200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.28 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=H100, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.28 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=B200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.27 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=B200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.26 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=B200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.26 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=H100, Baseline Method=Multinomial Sampling2026.03 | 1.25 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=B300, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.25 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=H200, Baseline Method=Multinomial Sampling2026.03 | 1.24 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=B200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.23 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=H100, Baseline Method=Multinomial Sampling2026.03 | 1.23 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=H200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.23 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=H100, Baseline Method=Multinomial Sampling2026.03 | 1.22 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=H200, Baseline Method=Multinomial Sampling2026.03 | 1.22 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=H200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.22 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=H100, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.22 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=H100, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.22 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=H100, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.21 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=H100, Baseline Method=Multinomial Sampling2026.03 | 1.21 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=H200, Baseline Method=Multinomial Sampling2026.03 | 1.21 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=H100, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.21 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=B200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.2 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=H100, Baseline Method=Multinomial Sampling2026.03 | 1.2 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=B200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.2 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=H100, Baseline Method=Multinomial Sampling2026.03 | 1.2 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=H200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.2 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=H200, Baseline Method=Multinomial Sampling2026.03 | 1.19 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=H200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.19 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=H200, Baseline Method=Multinomial Sampling2026.03 | 1.19 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=B300, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.18 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=H100, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.18 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=H100, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.18 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=B200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.18 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=H100, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.18 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=H200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.17 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=B300, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.17 | |
| FlashSamplingBatch Size (B)=64, GPU Architecture=H200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.17 | |
| FlashSamplingBatch Size (B)=256, GPU Architecture=B300, Baseline Method=Multinomial Sampling2026.03 | 1.17 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=H200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.16 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=B300, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.16 | |
| FlashSamplingBatch Size (B)=256, GPU Architecture=B200, Baseline Method=Multinomial Sampling2026.03 | 1.15 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=H100, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.14 | |
| FlashSamplingBatch Size (B)=32, GPU Architecture=H100, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.14 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=H100, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.14 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=B300, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.14 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=H100, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.13 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=B200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.13 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=B300, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.13 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=H100, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.13 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=B200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.13 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=B200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.13 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=B300, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.13 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=H100, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.12 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=H100, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.12 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=B300, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.12 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=B200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.12 | |
| FlashSamplingBatch Size (B)=16, GPU Architecture=H200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.11 | |
| FlashSamplingBatch Size (B)=1, GPU Architecture=H200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.09 | |
| FlashSamplingBatch Size (B)=4, GPU Architecture=H200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.09 | |
| FlashSamplingBatch Size (B)=8, GPU Architecture=H200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.09 | |
| FlashSamplingBatch Size (B)=2, GPU Architecture=H200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.08 | |
| FlashSamplingBatch Size (B)=256, GPU Architecture=B200, Baseline Method=FI1 (top-k/top-p)2026.03 | 1.08 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=H100, Baseline Method=FI2 (Gumbel-Max)2026.03 | 1.03 | |
| FlashSamplingBatch Size (B)=128, GPU Architecture=H200, Baseline Method=Multinomial Sampling2026.03 | 0.99 | |
| FlashSamplingBatch Size (B)=256, GPU Architecture=B200, Baseline Method=FI2 (Gumbel-Max)2026.03 | 0.92 | |
| FlashSamplingBatch Size (B)=256, GPU Architecture=B300, Baseline Method=FI2 (Gumbel-Max)2026.03 | 0.9 |