Quantization Latency Measurement on NVFP4 Operator
0.01Latency (ms)vllm default
Evaluation results
| Method | Links | ||
|---|---|---|---|
| vllm defaultBatch Size=128, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.01 | — | |
| ScaleSweepMSEBatch Size=128, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.01 | 0.97 | |
| vllm defaultBatch Size=256, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.012 | — | |
| ScaleSweepBatch Size=128, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.012 | 1.23 | |
| ScaleSweepMSEBatch Size=256, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.013 | 1.08 | |
| vllm defaultBatch Size=512, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.016 | — | |
| ScaleSweepBatch Size=256, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.016 | 1.35 | |
| ScaleSweepMSEBatch Size=512, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.017 | 1.03 | |
| ScaleSweepBatch Size=512, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.021 | 1.26 | |
| vllm defaultBatch Size=1024, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.025 | — | |
| ScaleSweepMSEBatch Size=1024, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.026 | 1.03 | |
| ScaleSweepBatch Size=1024, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.032 | 1.29 | |
| vllm defaultBatch Size=2048, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.041 | — | |
| ScaleSweepMSEBatch Size=2048, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.042 | 1.01 | |
| ScaleSweepBatch Size=2048, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.055 | 1.33 | |
| vllm defaultBatch Size=4096, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.072 | — | |
| ScaleSweepMSEBatch Size=4096, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.072 | 0.99 | |
| ScaleSweepBatch Size=4096, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.099 | 1.36 | |
| ScaleSweepMSEBatch Size=8192, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.134 | 1 | |
| vllm defaultBatch Size=8192, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.135 | — | |
| ScaleSweepBatch Size=8192, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.185 | 1.38 | |
| vllm defaultBatch Size=16384, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.258 | — | |
| ScaleSweepMSEBatch Size=16384, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.26 | 1.01 | |
| ScaleSweepBatch Size=16384, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.357 | 1.38 | |
| vllm defaultBatch Size=32768, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.502 | — | |
| ScaleSweepMSEBatch Size=32768, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.502 | 1 | |
| ScaleSweepBatch Size=32768, Hidden Dimension=8192, GPU=NVIDIA RTX PRO 6000 Blackwell2026.05 | 0.689 | 1.37 |