LLM Inference Performance on Synthetic Poisson Process Requests
18.7TPOT Speedup (%)FlashSampling
Evaluation Results
| Method | Links | |
|---|---|---|
| FlashSamplingB=64, Model=Qwen3-1.7B, Hardware=single B200 GPU2026.03 | 18.7 | |
| FlashSamplingB=16, Model=Qwen3-1.7B, Hardware=single B200 GPU2026.03 | 17.7 | |
| FlashSamplingB=8, Model=Qwen3-1.7B, Hardware=single B200 GPU2026.03 | 15.2 | |
| FlashSamplingB=4, Model=Qwen3-1.7B, Hardware=single B200 GPU2026.03 | 14.5 | |
| FlashSamplingB=2, Model=Qwen3-1.7B, Hardware=single B200 GPU2026.03 | 12.8 | |
| FlashSamplingB=1, Model=Qwen3-1.7B, Hardware=single B200 GPU2026.03 | 10.8 | |
| FlashSamplingB=32, Model=Qwen3-1.7B, Hardware=single B200 GPU2026.03 | 9.7 | |
| FlashSamplingB=2, Model=Qwen3-8B, Hardware=single B200 GPU2026.03 | 6.9 | |
| FlashSamplingB=64, Model=Qwen3-8B, Hardware=single B200 GPU2026.03 | 5.2 | |
| FlashSamplingB=32, Model=Qwen3-8B, Hardware=single B200 GPU2026.03 | 4.4 | |
| FlashSamplingB=4, Model=Qwen3-8B, Hardware=single B200 GPU2026.03 | 3.7 | |
| FlashSamplingB=8, Model=Qwen3-8B, Hardware=single B200 GPU2026.03 | 3.6 | |
| FlashSamplingB=16, Model=Qwen3-8B, Hardware=single B200 GPU2026.03 | 3 | |
| FlashSamplingB=1, Model=Qwen3-8B, Hardware=single B200 GPU2026.03 | 2.8 | |
| FlashSamplingB=4, Model=gpt-oss-120b, Hardware=single B200 GPU2026.03 | 2.4 | |
| FlashSamplingB=8, Model=gpt-oss-120b, Hardware=single B200 GPU2026.03 | 2.1 | |
| FlashSamplingB=1, Model=Qwen3-32B, Hardware=single B200 GPU2026.03 | 1.9 | |
| FlashSamplingB=2, Model=Qwen3-32B, Hardware=single B200 GPU2026.03 | 1.8 | |
| FlashSamplingB=16, Model=gpt-oss-120b, Hardware=single B200 GPU2026.03 | 1.8 | |
| FlashSamplingB=2, Model=gpt-oss-120b, Hardware=single B200 GPU2026.03 | 1.7 | |
| FlashSamplingB=4, Model=Qwen3-32B, Hardware=single B200 GPU2026.03 | 1.6 | |
| FlashSamplingB=64, Model=gpt-oss-120b, Hardware=single B200 GPU2026.03 | 1.6 | |
| FlashSamplingB=32, Model=gpt-oss-120b, Hardware=single B200 GPU2026.03 | 1.5 | |
| FlashSamplingB=32, Model=Qwen3-32B, Hardware=single B200 GPU2026.03 | 1.3 | |
| FlashSamplingB=64, Model=Qwen3-32B, Hardware=single B200 GPU2026.03 | 1.2 | |
| FlashSamplingB=8, Model=Qwen3-32B, Hardware=single B200 GPU2026.03 | 1.1 | |
| FlashSamplingB=16, Model=Qwen3-32B, Hardware=single B200 GPU2026.03 | 1.1 | |
| FlashSamplingB=1, Model=gpt-oss-120b, Hardware=single B200 GPU2026.03 | 0.3 |