Multimodal Evaluation on MM-Bench
83AccuracyCoLLaVO
Evaluation results
| Method | Links | |
|---|---|---|
| CoLLaVONumber of Parameters=7B, Setting=Zero-shot2024.02 | 83 | |
| MoAIZero-shot=true, Number of Parameters=7B2024.03 | 79.3 | |
| LLaVA-XTunerNumber of Parameters=20B, Setting=Zero-shot2024.02 | 75.1 | |
| LLaVA-XTunerZero-shot=true, Number of Parameters=20B2024.03 | 75.1 | |
| Intern-XCNumber of Parameters=7B, Setting=Zero-shot2024.02 | 74.4 | |
| Intern-XCZero-shot=true, Number of Parameters=7B2024.03 | 74.4 | |
| ShareGPT4VNumber of Parameters=7B, Setting=Zero-shot2024.02 | 68.8 | |
| ShareGPT4VZero-shot=true, Number of Parameters=7B2024.03 | 68.8 | |
| LLaVA1.5Number of Parameters=13B, Setting=Zero-shot2024.02 | 67.7 | |
| LLaVA1.5Zero-shot=true, Number of Parameters=13B2024.03 | 67.7 | |
| LLaVA-1.5-13B (Teacher)Bitwidth=BF162026.01 | 67.7 | |
| GRACEBitwidth=8-bit2026.01 | 66.1 | |
| CogVLMNumber of Parameters=17B, Setting=Zero-shot2024.02 | 65.8 | |
| CogVLMZero-shot=true, Number of Parameters=17B2024.03 | 65.8 | |
| LLaVoltaStages=Three, Scheme=compr. deeper, #Tokens=10597, Compression Ratio (CR)=174%, TFLOPS=8.26, Latency (ms)=68.5, Train Time=12.8h2024.06 | 65.6 | |
| LLaVoltaStages=Four, Scheme=deeper then wider, #Tokens=10863, Compression Ratio (CR)=170%, TFLOPS=8.26, Latency (ms)=68.5, Train Time=12.8h2024.06 | 65.6 | |
| LLaVoltaStages=Single, Scheme=no compression, #Tokens=18432, Compression Ratio (CR)=-, TFLOPS=8.26, Latency (ms)=68.5, Train Time=15.3h2024.06 | 65.3 | |
| LLaVoltaStages=Four, Scheme=wider then deeper, #Tokens=11088, Compression Ratio (CR)=166%, TFLOPS=8.26, Latency (ms)=68.5, Train Time=12.9h2024.06 | 65.3 | |
| LLaVoltaStages=Three, Scheme=last stage compression, #Tokens=7848, Compression Ratio (CR)=235%, TFLOPS=5.47, Latency (ms)=52.2, Train Time=12.4h2024.06 | 65.3 | |
| GRACEBitwidth=4-bit2026.01 | 65 | |
| LLaVoltaStages=Two, Scheme=compression, #Tokens=10062, Compression Ratio (CR)=183%, TFLOPS=8.26, Latency (ms)=68.5, Train Time=12.8h2024.06 | 64.8 | |
| LLaVoltaStages=Three, Scheme=compr. wider, #Tokens=10407, Compression Ratio (CR)=177%, TFLOPS=8.26, Latency (ms)=68.5, Train Time=12.8h2024.06 | 64.8 | |
| LLaVA-1.5 + PADL+ (8×)LLM=Vicuna-7B, Compression Method=PADL, Compression Ratio=8x, Adapter fine-tuning=true2026.06 | 64.7 | |
| mPLUG-Owl2Number of Parameters=7B, Setting=Zero-shot2024.02 | 64.5 | |
| mPLUG-Owl2Zero-shot=true, Number of Parameters=7B2024.03 | 64.5 | |
| LLaVA1.5Number of Parameters=7B, Setting=Zero-shot2024.02 | 64.3 | |
| LLaVA1.5Zero-shot=true, Number of Parameters=7B2024.03 | 64.3 | |
| LLaVA-v1.5LLM=Vicuna-7B2023.12 | 64.3 | |
| LLaVA-1.5-7B (Baseline)Bitwidth=BF162026.01 | 64.3 | |
| LLaVA-1.5LLM=Vicuna-7B2026.06 | 64.3 | |
| LLaVA-1.5 + PADL (8×)LLM=Vicuna-7B, Compression Method=PADL, Compression Ratio=8x, Adapter fine-tuning=false2026.06 | 64.1 | |
| QATBitwidth=4-bit2026.01 | 63.9 | |
| LLaVA-1.5 + FreePrunerLLM=Vicuna-7B, Compression Method=FreePruner2026.06 | 63.8 | |
| AWQBitwidth=4-bit2026.01 | 63.1 | |
| LLaVA-1.5 + PruMergeLLM=Vicuna-7B, Compression Method=PruMerge2026.06 | 62.9 | |
| RTNBitwidth=4-bit2026.01 | 62.6 | |
| Qwen-VL-ChatNumber of Parameters=7B, Setting=Zero-shot2024.02 | 60.6 | |
| Qwen-VL-ChatZero-shot=true, Number of Parameters=7B2024.03 | 60.6 | |
| Qwen-VLLLM=Qwen-7B2023.12 | 60.6 | |
| OneLLM-7BLLM=LLAMA2-7B2023.12 | 60 | |
| ShikraNumber of Parameters=13B, Setting=Zero-shot2024.02 | 58.8 | |
| ShikraZero-shot=true, Number of Parameters=13B2024.03 | 58.8 | |
| ShikraLLM=Vicuna-13B2026.06 | 58.8 | |
| IDEFICSNumber of Parameters=80B, Setting=Zero-shot2024.02 | 54.5 | |
| IDEFICSZero-shot=true, Number of Parameters=80B2024.03 | 54.5 | |
| IDEFICS-80BLLM=LLaMA-65B2023.12 | 54.5 | |
| IDEFICS-80BLLM=LLaMA-65B2026.06 | 54.5 | |
| OtterNumber of Parameters=7B, Setting=Zero-shot2024.02 | 48.3 | |
| OtterZero-shot=true, Number of Parameters=7B2024.03 | 48.3 | |
| IDEFICSNumber of Parameters=9B, Setting=Zero-shot2024.02 | 48.2 | |
| IDEFICSZero-shot=true, Number of Parameters=9B2024.03 | 48.2 | |
| IDEFICS-9BLLM=LLaMA-7B2023.12 | 48.2 | |
| IDEFICS-9BLLM=LLaMA-7B2026.06 | 48.2 | |
| mPLUG-OwlNumber of Parameters=7B, Setting=Zero-shot2024.02 | 46.6 | |
| mPLUG-OwlZero-shot=true, Number of Parameters=7B2024.03 | 46.6 | |
| LLaMA-Ad.v2LLM=LLaMA-7B2023.12 | 38.9 | |
| Qwen-VLNumber of Parameters=7B, Setting=Zero-shot2024.02 | 38.2 | |
| Qwen-VLZero-shot=true, Number of Parameters=7B2024.03 | 38.2 | |
| Qwen-VLLLM=Qwen-7B2026.06 | 38.2 | |
| InstructBLIPNumber of Parameters=7B, Setting=Zero-shot2024.02 | 36 | |
| InstructBLIPZero-shot=true, Number of Parameters=7B2024.03 | 36 | |
| InstructBLIPLLM=Vicuna-7B2023.12 | 36 | |
| InstructBLIPLLM=Vicuna-7B2026.06 | 36 | |
| LLaVANumber of Parameters=7B, Setting=Zero-shot2024.02 | 34.1 | |
| LLaVAZero-shot=true, Number of Parameters=7B2024.03 | 34.1 | |
| MiniGPT-4Number of Parameters=7B, Setting=Zero-shot2024.02 | 23 | |
| MiniGPT-4Zero-shot=true, Number of Parameters=7B2024.03 | 23 |