LLM Generation on RTX 3090 24GB (inference)
1,150Max Batch SizeLayer-Condensed KV Cache
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Layer-Condensed KV CacheModel Size=1.1B, Seq. Length=5+2043, w=22024.05 | 1,150 | 10,033.4 | |
| Layer-Condensed KV CacheModel Size=1.1B, Seq. Length=5+8187, w=22024.05 | 384 | 4,113.37 | |
| Layer-Condensed KV CacheModel Size=1.1B, Seq. Length=5+2043, w=102024.05 | 289 | 7,239.92 | |
| LlamaModel Size=1.1B, Seq. Length=5+20432024.05 | 239 | 5,142.86 | |
| Layer-Condensed KV CacheModel Size=1.1B, Seq. Length=5+8187, w=102024.05 | 119 | 2,374.05 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=512+512, w=22024.05 | 95 | 378.89 | |
| Layer-Condensed KV CacheModel Size=30B, CPU-offload=true, Seq. Length=512+1024, w=22024.05 | 83 | 5.99 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=512+1024, w=22024.05 | 72 | 401.92 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=5+2043, w=22024.05 | 64 | 534.02 | |
| LlamaModel Size=1.1B, Seq. Length=5+81872024.05 | 48 | 1,424.96 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=512+512, w=102024.05 | 32 | 380.6 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=2048+2048, w=22024.05 | 23 | 171.65 | |
| Layer-Condensed KV CacheModel Size=30B, CPU-offload=true, Seq. Length=512+1024, w=102024.05 | 23 | 1.63 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=5+2043, w=102024.05 | 16 | 315.38 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=512+1024, w=102024.05 | 16 | 310.05 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=5+8187, w=22024.05 | 12 | 151.91 | |
| LlamaModel Size=7B, Seq. Length=512+5122024.05 | 9 | 225.31 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=2048+2048, w=102024.05 | 8 | 119.68 | |
| LlamaModel Size=7B, Seq. Length=512+10242024.05 | 7 | 174.11 | |
| LlamaModel Size=7B, Seq. Length=5+20432024.05 | 5 | 140.88 | |
| Layer-Condensed KV CacheModel Size=7B, Seq. Length=5+8187, w=102024.05 | 4 | 83.8 | |
| LlamaModel Size=30B, CPU-offload=true, Seq. Length=512+10242024.05 | 4 | 0.23 | |
| LlamaModel Size=7B, Seq. Length=2048+20482024.05 | 2 | 56.98 | |
| LlamaModel Size=7B, Seq. Length=5+81872024.05 | 1 | 32.02 |