Inference Speedup on BERT base
21.5SpeedupTRT FP16
Evaluation results
| Method | Links | |
|---|---|---|
| TRT FP16Precision=FP16, Approach=Hybrid precision pipeline, Batch Size=1, Sequence Length=1282026.03 | 21.5 | |
| TRT FP32Precision=FP32, Approach=Graph opt + auto-tuning, Batch Size=1, Sequence Length=1282026.03 | 10.8 | |
| FasterTFPrecision=FP16, Approach=Custom CUDA kernels, Batch Size=1, Sequence Length=1282026.03 | 4 | |
| FastFormersPrecision=FP32, Approach=Operator-specific kernels, Batch Size=1, Sequence Length=1282026.03 | 1.8 | |
| PyTorch GPUPrecision=FP32, Approach=Eager-mode baseline, Batch Size=1, Sequence Length=1282026.03 | 1 |