Inference Throughput on BERT, GPT-2, and OPT (BS=2, SL=256)
57,117.4Original Throughput (tokens/s)BERT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| BERTHardware=A100, Batch size=2, Sequence length=256, Number of runs=5, Measured inferences=50, Warm-up steps=502026.05 | 57,117.4 | 58,756 | 1,638.7 | |
| GPT-2Hardware=A100, Batch size=2, Sequence length=256, Number of runs=5, Measured inferences=50, Warm-up steps=502026.05 | 55,351.4 | 57,787.8 | 2,436.5 | |
| OPTHardware=A100, Batch size=2, Sequence length=256, Number of runs=5, Measured inferences=50, Warm-up steps=502026.05 | 54,468.1 | 57,853.1 | 3,385 | |
| BERTHardware=V100, Batch size=2, Sequence length=256, Number of runs=5, Measured inferences=50, Warm-up steps=502026.05 | 44,329 | 46,126.1 | 1,797.1 | |
| OPTHardware=V100, Batch size=2, Sequence length=256, Number of runs=5, Measured inferences=50, Warm-up steps=502026.05 | 43,144.9 | 44,263.9 | 1,119 | |
| GPT-2Hardware=V100, Batch size=2, Sequence length=256, Number of runs=5, Measured inferences=50, Warm-up steps=502026.05 | 42,199 | 43,760.7 | 1,561.7 |