Multimodal Reasoning on MMBench (dev)
87.6AccuracyGPT-4o
Evaluation Results
| Method | Links | |
|---|---|---|
| GPT-4oindependent reproduction=true, IoT prompting=true2024.05 | 87.6 | |
| GPT-4oindependent reproduction=true, IoT prompting=false2024.05 | 86.2 | |
| Gemini-Pro-1.5independent reproduction=true, IoT prompting=true2024.05 | 79.6 | |
| InternLM-XComposer2-VLParams=7B2024.05 | 79.5 | |
| Qwen-VL-Max2024.05 | 78.1 | |
| LLaVA-FA-7BLLM=InternLM-2-20B, #Sample=5M, #Param=≥7B2026.01 | 74.5 | |
| Deepseek-VL-7BLLM=DLLM-7B, #Sample=2000M, #Param=≥7B2026.01 | 74.1 | |
| Imp-3BLLM=Phi-2-2.7B, #Sample=1.6M, #Param=∼3B2026.01 | 72.9 | |
| Gemini-Pro-1.5independent reproduction=true, IoT prompting=false2024.05 | 71 | |
| LLaVA-FA-3BLLM=LLaMA-3-8B, #Sample=5M, #Param=∼3B2026.01 | 70.5 | |
| LLaVA-NeXTLLM=Vicuna-1.5-13B, #Sample=1.3M, #Param=≥7B2026.01 | 70.4 | |
| LLaVA-v1.5-13BParams=13.4B2024.05 | 69.2 | |
| VILA-7BLLM=LLaMA-7B, #Sample=50M, #Param=≥7B2026.01 | 68.9 | |
| Bunny-3BLLM=Phi-2-2.7B, #Sample=2.7M, #Param=∼3B2026.01 | 68.6 | |
| MiniCPM-V-2LLM=MiniCPM-2.4B, #Sample=570M, #Param=∼3B2026.01 | 68.5 | |
| LLaVA-FA-2BLLM=Qwen-2.5-7B, #Sample=5M, #Param=∼2B2026.01 | 66.7 | |
| Qwen-VL-Plus2024.05 | 66.2 | |
| MoE-LLaVA-3BLLM=Phi-2-2.7B, #Sample=2.2M, #Param=∼3B2026.01 | 65.2 | |
| DeepSeek-VL-1.3BLLM=DLLM-1.3B, #Sample=2000M, #Param=∼2B2026.01 | 64.6 | |
| LLaVA-1.5LLM=Vicuna-7B, Resolution (Res.)=336, Pre-training data size (PT)=558K, Instruction tuning data size (IT)=665K2023.12 | 64.3 | |
| LLaVA-1.5-7BLLM=Vicuna-1.5-7B, #Sample=1.2M, #Param=≥7B2026.01 | 64.3 | |
| MiniCPM-VLLM=MiniCPM-2.4B, #Sample=570M, #Param=∼3B2026.01 | 64 | |
| Imp-2BLLM=Qwen-1.5-1.8B, #Sample=1.6M, #Param=∼2B2026.01 | 63.8 | |
| CogVLMLLM=Vicuna-7B, #Sample=1500M, #Param=≥7B2026.01 | 63.7 | |
| VILA-3BLLM=LLaMA-2.7B, #Sample=51M, #Param=∼3B2026.01 | 63.4 | |
| MobileVLMv2LLM=MLLaMA-2.7B, #Sample=3.6M, #Param=∼3B2026.01 | 63.2 | |
| Qwen-VL-ChatLLM=Qwen-7B, #Sample=1450M, #Param=≥7B2026.01 | 60.6 | |
| Mini-Gemini-2BLLM=Gemma-2B, #Sample=2.7M, #Param=∼2B2026.01 | 59.8 | |
| MoE-LLaVA-2BLLM=Qwen-1.5-1.8B, #Sample=2.2M, #Param=∼2B2026.01 | 59.7 | |
| MobileVLM 3BLLM=MobileLLaMA 2.7B, Resolution (Res.)=336, Pre-training data size (PT)=558K, Instruction tuning data size (IT)=665K2023.12 | 59.6 | |
| MobileVLMLLM=MLLaMA-2.7B, #Sample=1.3M, #Param=∼3B2026.01 | 59.6 | |
| Bunny-2BLLM=Qwen-1.5-1.8B, #Sample=2.7M, #Param=∼2B2026.01 | 59.1 | |
| ShikraLLM=Vicuna-13B, Resolution (Res.)=224, Pre-training data size (PT)=600K, Instruction tuning data size (IT)=5.5M2023.12 | 58.8 | |
| LLaVA-FA-1BLLM=Qwen-2.5-3B, #Sample=5M, #Param=∼1B2026.01 | 58.3 | |
| MobileVLM 3B w/ LORALLM=MobileLLaMA 2.7B, Resolution (Res.)=336, Pre-training data size (PT)=558K, Instruction tuning data size (IT)=665K2023.12 | 57 | |
| SPHINX-TinyLLM=TLLaMA-1.1B, #Sample=15M, #Param=∼1B2026.01 | 56.6 | |
| IDEFICS-80BLLM=LLaMA-65B, Resolution (Res.)=224, Pre-training data size (PT)=353M, Instruction tuning data size (IT)=1M2023.12 | 54.5 | |
| MobileVLM 1.7BLLM=MobileLLaMA 1.4B, Resolution (Res.)=336, Pre-training data size (PT)=558K, Instruction tuning data size (IT)=665K2023.12 | 53.2 | |
| MobileVLM 1.7B w/ LORALLM=MobileLLaMA 1.4B, Resolution (Res.)=336, Pre-training data size (PT)=558K, Instruction tuning data size (IT)=665K2023.12 | 50.4 | |
| mPLUG-OwlLLM=LLaMA-7B, Resolution (Res.)=224, Pre-training data size (PT)=2.1M, Instruction tuning data size (IT)=102K2023.12 | 49.4 | |
| IDEFICS-9BLLM=LLaMA-7B, Resolution (Res.)=224, Pre-training data size (PT)=353M, Instruction tuning data size (IT)=1M2023.12 | 48.2 | |
| Qwen-VLLLM=Qwen-7B, Resolution (Res.)=448, Pre-training data size (PT)=1.4B, Instruction tuning data size (IT)=50M2023.12 | 38.2 | |
| InstructBLIPLLM=Vicuna-7B, Resolution (Res.)=224, Pre-training data size (PT)=129M, Instruction tuning data size (IT)=1.2M2023.12 | 36 | |
| MiniGPT-4Params=8B2024.05 | 24.3 | |
| MiniGPT-4LLM=Vicuna-7B, Resolution (Res.)=224, Pre-training data size (PT)=5M, Instruction tuning data size (IT)=5K2023.12 | 23 | |
| MiniGPT-v2LLM=LLaMA-7B, Resolution (Res.)=448, Pre-training data size (PT)=23M, Instruction tuning data size (IT)=1M2023.12 | 12.2 | |
| OpenflamingoLLM=MPT-7B, Resolution (Res.)=336, Pre-training data size (PT)=180M, Instruction tuning data size (IT)=-2023.12 | 4.6 |