VLM Inference Latency on Jetson Orin llama.cpp quantization
2.89VE Latency (ms/patch)LLaVA-v1.5-336
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| LLaVA-v1.5-336Language Model=Vicuna 7B, Size (GB)=4.70, Quantization=4-bit, Input Resolution=336x3362023.12 | 2.89 | 9,281 | 367.26 | 17.74 | 19.75 | |
| LLaVA-v1.5-336Language Model=OpenLLaMA 3B, Size (GB)=2.88, Quantization=4-bit, Input Resolution=336x3362023.12 | 2.94 | 22,270 | 474.49 | 30.66 | 12.52 | |
| LLaVA-v1.5-336Language Model=TinyLLaMA 1B, Size (GB)=1.18, Quantization=4-bit, Input Resolution=336x3362023.12 | 2.98 | 24,655 | 1,253.94 | 76.63 | 5.9 | |
| MobileVLM-336Language Model=MobileLLaMA 2.7B, Size (GB)=2.14, Quantization=4-bit, Input Resolution=336x3362023.12 | 3.11 | 15,678 | 440.6 | 38.34 | 8.31 | |
| MobileVLM-336Language Model=MobileLLaMA 1.4B, Size (GB)=1.40, Quantization=4-bit, Input Resolution=336x3362023.12 | 3.32 | 17,712 | 667.69 | 65.27 | 5.14 |