E2E Generation Latency on Server-side E2E Generation L=2K+2K prefix+decode (test)
55.1Latency (s)TransAct-2.6B
Evaluation Results
| Method | Links | |
|---|---|---|
| TransAct-2.6BBatch Size (B)=1, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=2K+2K2024.07 | 55.1 | |
| Sheared-LLaMA-2.7BBatch Size (B)=1, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=2K+2K2024.07 | 56.5 | |
| LLaMA2-7BBatch Size (B)=1, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=2K+2K2024.07 | 57 | |
| TransAct-2.6BBatch Size (B)=4, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=2K+2K2024.07 | 58.4 | |
| Sheared-LLaMA-2.7BBatch Size (B)=4, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=2K+2K2024.07 | 59.7 | |
| LLaMA2-7BBatch Size (B)=4, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=2K+2K2024.07 | 84 | |
| TransAct-2.6BBatch Size (B)=16, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=2K+2K2024.07 | 129 | |
| Sheared-LLaMA-2.7BBatch Size (B)=16, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=2K+2K2024.07 | 156.6 |