Server-side E2E Generation Latency (L=1K+2K prefix+decode, test)
55E2E Generation Latency (s)TransAct-2.6B
Evaluation Results
| Method | Links | |
|---|---|---|
| TransAct-2.6BBatch Size (B)=1, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07 | 55 | |
| Sheared-LLaMA-2.7BBatch Size (B)=1, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07 | 56.2 | |
| LLaMA2-7BBatch Size (B)=1, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07 | 56.9 | |
| TransAct-2.6BBatch Size (B)=4, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07 | 57.1 | |
| Sheared-LLaMA-2.7BBatch Size (B)=4, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07 | 57.5 | |
| LLaMA2-7BBatch Size (B)=4, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07 | 67 | |
| TransAct-2.6BBatch Size (B)=16, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07 | 95 | |
| Sheared-LLaMA-2.7BBatch Size (B)=16, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07 | 112.1 | |
| LLaMA2-7BBatch Size (B)=16, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07 | 177.2 |