List your company
ResearchFormerly WizWand

Server-side E2E Generation Latency (L=1K+2K prefix+decode, test)

55E2E Generation Latency (s)TransAct-2.6B
50.11283.106116.1149.094Jul 8, 2024
Updated 8mo ago

Evaluation results

MethodLinks
TransAct-2.6BBatch Size (B)=1, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.0755
Sheared-LLaMA-2.7BBatch Size (B)=1, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.0756.2
LLaMA2-7BBatch Size (B)=1, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.0756.9
TransAct-2.6BBatch Size (B)=4, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.0757.1
Sheared-LLaMA-2.7BBatch Size (B)=4, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.0757.5
LLaMA2-7BBatch Size (B)=4, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.0767
TransAct-2.6BBatch Size (B)=16, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.0795
Sheared-LLaMA-2.7BBatch Size (B)=16, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07112.1
LLaMA2-7BBatch Size (B)=16, GPU=NVIDIA A100, Precision=bfloat16, Context Length (L)=1K+2K2024.07177.2