Inference Latency on GPT-J-6B
0.203Original Inference Time (s)GPT-J-6B (Standard)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| GPT-J-6B (Standard)Batch Size=4, Seq Length=642026.05 | 0.203 | — | — | |
| GPT-J-6B (Standard)Batch Size=8, Seq Length=642026.05 | 0.4187 | — | — | |
| GPT-J-6B (Standard)Batch Size=16, Seq Length=642026.05 | 0.7793 | — | — | |
| GPT-J-6B (Standard)Batch Size=4, Seq Length=2562026.05 | 0.784 | — | — | |
| GPT-J-6B (Standard)Batch Size=8, Seq Length=2562026.05 | 1.5709 | — | — | |
| GPT-J-6B (Standard)Batch Size=16, Seq Length=2562026.05 | 2.9893 | — | — | |
| GPT-J-6B (Standard)Batch Size=4, Seq Length=10242026.05 | 3.0976 | — | — | |
| GPT-J-6B (Standard)Batch Size=8, Seq Length=10242026.05 | 6.2038 | — | — | |
| CBWC+RMSNormBatch Size=4, Seq Length=642026.05 | — | 0.1931 | 4.89 | |
| CBWC+RMSNormBatch Size=4, Seq Length=2562026.05 | — | 0.7453 | 4.94 | |
| CBWC+RMSNormBatch Size=4, Seq Length=10242026.05 | — | 2.8637 | 7.55 | |
| CBWC+RMSNormBatch Size=8, Seq Length=642026.05 | — | 0.401 | 4.22 | |
| CBWC+RMSNormBatch Size=8, Seq Length=2562026.05 | — | 1.4666 | 6.64 | |
| CBWC+RMSNormBatch Size=8, Seq Length=10242026.05 | — | 5.7415 | 7.45 | |
| CBWC+RMSNormBatch Size=16, Seq Length=642026.05 | — | 0.7409 | 4.92 | |
| CBWC+RMSNormBatch Size=16, Seq Length=2562026.05 | — | 2.7601 | 7.67 |