Inference Throughput on GPT-J 6B
1,484Inference Throughput (tokens/s)GPT-J-6B (Proposed Method)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| GPT-J-6B (Proposed Method)Batch Size=16, Seq Length=256, Hardware=single A100-40GB GPU2026.05 | 1,484 | 113.8 | |
| GPT-J-6B (Proposed Method)Batch Size=4, Seq Length=1024, Hardware=single A100-40GB GPU2026.05 | 1,430.3 | 108 | |
| GPT-J-6B (Proposed Method)Batch Size=8, Seq Length=1024, Hardware=single A100-40GB GPU2026.05 | 1,426.8 | 106.3 | |
| GPT-J-6B (Proposed Method)Batch Size=8, Seq Length=256, Hardware=single A100-40GB GPU2026.05 | 1,396.4 | 92.7 | |
| GPT-J-6B (Proposed Method)Batch Size=16, Seq Length=64, Hardware=single A100-40GB GPU2026.05 | 1,382 | 68 | |
| GPT-J-6B (Proposed Method)Batch Size=4, Seq Length=256, Hardware=single A100-40GB GPU2026.05 | 1,374 | 67.9 | |
| GPT-J-6BBatch Size=16, Seq Length=256, Hardware=single A100-40GB GPU2026.05 | 1,370.2 | — | |
| GPT-J-6B (Proposed Method)Batch Size=4, Seq Length=64, Hardware=single A100-40GB GPU2026.05 | 1,325.7 | 64.8 | |
| GPT-J-6BBatch Size=4, Seq Length=1024, Hardware=single A100-40GB GPU2026.05 | 1,322.3 | — | |
| GPT-J-6BBatch Size=8, Seq Length=1024, Hardware=single A100-40GB GPU2026.05 | 1,320.5 | — | |
| GPT-J-6BBatch Size=16, Seq Length=64, Hardware=single A100-40GB GPU2026.05 | 1,314.1 | — | |
| GPT-J-6BBatch Size=4, Seq Length=256, Hardware=single A100-40GB GPU2026.05 | 1,306.1 | — | |
| GPT-J-6BBatch Size=8, Seq Length=256, Hardware=single A100-40GB GPU2026.05 | 1,303.8 | — | |
| GPT-J-6B (Proposed Method)Batch Size=8, Seq Length=64, Hardware=single A100-40GB GPU2026.05 | 1,276.8 | 53.8 | |
| GPT-J-6BBatch Size=4, Seq Length=64, Hardware=single A100-40GB GPU2026.05 | 1,260.9 | — | |
| GPT-J-6BBatch Size=8, Seq Length=64, Hardware=single A100-40GB GPU2026.05 | 1,223 | — |