Text Generation on General Generation Workload (1000 tokens latency)
72.51Prefill Latency (ms)Dynamic-LLaVA-13B|T
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Dynamic-LLaVA-13B|TBatch size=1, Total generation tokens=1000, Hardware=A100 (80G), KV cache=Enabled2024.12 | 72.51 | 26.85 | |
| LLaVA-1.5-13B+H2OBatch size=1, Total generation tokens=1000, Hardware=A100 (80G), KV cache=Enabled, r=0.52024.12 | 111.84 | 37.71 | |
| LLaVA-1.5-13BBatch size=1, Total generation tokens=1000, Hardware=A100 (80G), KV cache=Enabled2024.12 | 124.52 | 28.42 |