List your company
ResearchFormerly WizWand

LLM Generation on RTX 3090 24GB (inference)

1,150Max Batch SizeLayer-Condensed KV Cache
-44.96265.27575.5885.73May 17, 2024
Updated 8mo ago

Evaluation results

MethodLinks
Layer-Condensed KV CacheModel Size=1.1B, Seq. Length=5+2043, w=22024.051,15010,033.4
Layer-Condensed KV CacheModel Size=1.1B, Seq. Length=5+8187, w=22024.053844,113.37
Layer-Condensed KV CacheModel Size=1.1B, Seq. Length=5+2043, w=102024.052897,239.92
LlamaModel Size=1.1B, Seq. Length=5+20432024.052395,142.86
Layer-Condensed KV CacheModel Size=1.1B, Seq. Length=5+8187, w=102024.051192,374.05
Layer-Condensed KV CacheModel Size=7B, Seq. Length=512+512, w=22024.0595378.89
Layer-Condensed KV CacheModel Size=30B, CPU-offload=true, Seq. Length=512+1024, w=22024.05835.99
Layer-Condensed KV CacheModel Size=7B, Seq. Length=512+1024, w=22024.0572401.92
Layer-Condensed KV CacheModel Size=7B, Seq. Length=5+2043, w=22024.0564534.02
LlamaModel Size=1.1B, Seq. Length=5+81872024.05481,424.96
Layer-Condensed KV CacheModel Size=7B, Seq. Length=512+512, w=102024.0532380.6
Layer-Condensed KV CacheModel Size=7B, Seq. Length=2048+2048, w=22024.0523171.65
Layer-Condensed KV CacheModel Size=30B, CPU-offload=true, Seq. Length=512+1024, w=102024.05231.63
Layer-Condensed KV CacheModel Size=7B, Seq. Length=5+2043, w=102024.0516315.38
Layer-Condensed KV CacheModel Size=7B, Seq. Length=512+1024, w=102024.0516310.05
Layer-Condensed KV CacheModel Size=7B, Seq. Length=5+8187, w=22024.0512151.91
LlamaModel Size=7B, Seq. Length=512+5122024.059225.31
Layer-Condensed KV CacheModel Size=7B, Seq. Length=2048+2048, w=102024.058119.68
LlamaModel Size=7B, Seq. Length=512+10242024.057174.11
LlamaModel Size=7B, Seq. Length=5+20432024.055140.88
Layer-Condensed KV CacheModel Size=7B, Seq. Length=5+8187, w=102024.05483.8
LlamaModel Size=30B, CPU-offload=true, Seq. Length=512+10242024.0540.23
LlamaModel Size=7B, Seq. Length=2048+20482024.05256.98
LlamaModel Size=7B, Seq. Length=5+81872024.05132.02