Inference Latency on Synthetic Sequence length 256
0.009Original Latency (s)BERT
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| BERTHardware=A100, Batch Size=2, Sequence Length=2562026.05 | 0.009 | — | — | |
| CBWCModel=BERT, Hardware=A100, Batch Size=2, Sequence Length=2562026.05 | 0.009 | 0.0087 | 2.79 | |
| GPT-2Hardware=A100, Batch Size=2, Sequence Length=2562026.05 | 0.0093 | — | — | |
| CBWCModel=GPT-2, Hardware=A100, Batch Size=2, Sequence Length=2562026.05 | 0.0093 | 0.0089 | 4.22 | |
| OPTHardware=A100, Batch Size=2, Sequence Length=2562026.05 | 0.0094 | — | — | |
| CBWCModel=OPT, Hardware=A100, Batch Size=2, Sequence Length=2562026.05 | 0.0094 | 0.0089 | 5.85 | |
| BERTHardware=V100, Batch Size=2, Sequence Length=2562026.05 | 0.0116 | — | — | |
| CBWCModel=BERT, Hardware=V100, Batch Size=2, Sequence Length=2562026.05 | 0.0116 | 0.0111 | 3.9 | |
| OPTHardware=V100, Batch Size=2, Sequence Length=2562026.05 | 0.0119 | — | — | |
| CBWCModel=OPT, Hardware=V100, Batch Size=2, Sequence Length=2562026.05 | 0.0119 | 0.0116 | 2.53 | |
| GPT-2Hardware=V100, Batch Size=2, Sequence Length=2562026.05 | 0.0121 | — | — | |
| CBWCModel=GPT-2, Hardware=V100, Batch Size=2, Sequence Length=2562026.05 | 0.0121 | 0.0117 | 3.57 |