ResearchBenchmarksInference Latency on VLM decode averageInference Latency on VLM decode averageFollow21.1Latency (ms)W3A1620.7623.05525.3527.645Dec 27, 2024Evaluation resultsMethodMethodLinksLatency (ms)W3A16Model=LLaVA-onevision-...Model=LLaVA-onevision-7B, Hardware=RTX 4090, Quantization=W3A162024.1221.1W4A8Model=LLaVA-onevision-...Model=LLaVA-onevision-7B, Hardware=RTX 4090, Quantization=W4A82024.1226.3FP16Model=LLaVA-onevision-...Model=LLaVA-onevision-7B, Hardware=RTX 4090, Quantization=FP162024.1229.6