Compositional Reasoning on Compositional Reasoning Suite Aggregated
93.1Sugarcrepe ScoreLLaVA-7B†
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| LLaVA-7B†Size=~7B, LLM=Qwen2.5-7B, # Samples=1.2 M2025.10 | 93.1 | — | 82.9 | 68.3 | 68.2 | 78.1 | |
| MiniCPM-VSize=~4B, LLM=MiniCPM-2.4B, # Samples=570 M2025.10 | 90 | — | 70.6 | 65.3 | 55.3 | 70.3 | |
| Qwen2.5-VL-7BSize=~7B, LLM=Qwen2-7B, # Samples=1500 M2025.10 | 88.5 | — | 77.4 | 68.4 | 75.3 | 77.4 | |
| LLaVA-7B†Size=~7B, LLM=Qwen1.5-7B, # Samples=1.2 M2025.10 | 87.3 | — | 78.5 | 65.7 | 58.4 | 72.5 | |
| Deepseek-VL-7BSize=~7B, LLM=DLLM-7B, # Samples=2000 M2025.10 | 86.2 | — | 78.8 | 66.2 | 61.7 | 73.2 | |
| LLaVA-4B† (Teacher)Size=~4B, LLM=Qwen1.5-4B, # Samples=1.2 M2025.10 | 83 | — | 75.5 | 64.8 | 57.8 | 70.3 | |
| CompoDistill-2BSize=~2B, LLM=Qwen1.5-1.8B, # Samples=1.2 M2025.10 | 82.9 | — | 69.4 | 63.3 | 51.2 | 66.7 | |
| CogVLM-7BSize=~7B, LLM=Vicuna-7B, # Samples=1500 M2025.10 | 81.8 | — | 63.2 | 64.5 | 57.2 | 66.7 | |
| MoE-LLaVA-2BSize=~2B, LLM=Qwen1.5-1.8B, # Samples=2.2 M2025.10 | 80.8 | — | 62.5 | 62 | 50.6 | 63.9 | |
| MoE-LLaVA-3BSize=~4B, LLM=Phi2-2.7B, # Samples=2.6 M2025.10 | 80.5 | — | 70.4 | 64.4 | 54.1 | 67.3 | |
| Imp-3BSize=~4B, LLM=Phi2-2.7B, # Samples=1.5 M2025.10 | 78.1 | — | 61.1 | 59.5 | 38.3 | 59.3 | |
| LLaVADI-2B†Size=~2B, LLM=Qwen1.5-1.8B, # Samples=1.2 M2025.10 | 76.9 | — | 61.5 | 54.1 | 48.4 | 60.2 | |
| LLaVA-MoD-2B†Size=~2B, LLM=Qwen1.5-1.8B, # Samples=5.0 M2025.10 | 76.9 | — | 63.8 | 60.3 | 49.4 | 62.6 | |
| Bunny-3BSize=~4B, LLM=Phi2-2.7B, # Samples=2.6 M2025.10 | 75.8 | — | 67.2 | 59.7 | 53.1 | 64 | |
| LLaVA-KD-2BSize=~2B, LLM=Qwen1.5-1.8B, # Samples=1.2 M2025.10 | 75.3 | — | 63.6 | 57.9 | 49.3 | 61.5 | |
| LLaVA-2B (SFT model)Size=~2B, LLM=Qwen1.5-1.8B, # Samples=1.2 M2025.10 | 73.3 | — | 63.9 | 58.5 | 47.2 | 60.7 | |
| Imp-2BSize=~2B, LLM=Qwen1.5-1.8B, # Samples=1.5 M2025.10 | 71 | — | 59.6 | 58.5 | 51.1 | 60.1 | |
| MobileVLM-1.7BSize=~2B, LLM=MobileLLaMA-1.4B, # Samples=1.2 M2025.10 | 69.8 | — | 64 | 53.8 | 43.4 | 57.7 | |
| Bunny-2BSize=~2B, LLM=Qwen1.5-1.8B, # Samples=2.6 M2025.10 | 68.6 | — | 66.5 | 57.8 | 48.5 | 60.3 | |
| MobileVLM-3BSize=~4B, LLM=MobileLLaMA-2.7B, # Samples=1.3 M2025.10 | 67.9 | — | 72.1 | 57.2 | 43.9 | 60.3 | |
| MiniGemini-2BSize=~2B, LLM=Gemma-2B, # Samples=2.7 M2025.10 | 67 | — | 62 | 58 | 50.5 | 59.4 | |
| Qwen2.5-VL-3BSize=~4B, LLM=Qwen2-3B, # Samples=1.2 M2025.10 | 57.1 | — | 65.9 | 67 | 63.5 | 63.4 | |
| Deepseek-VL-1.3BSize=~2B, LLM=DLLM-1.3B, # Samples=2000 M2025.10 | 36.6 | — | 38.6 | 39.9 | 60.8 | 44 | |
| LaCLIPPre-training Dataset=CC3M2024.11 | — | 23.65 | — | — | — | — | |
| LaCLIPPre-training Dataset=CC12M2024.11 | — | 25.96 | — | — | — | — | |
| LaCLIP + HNPre-training Dataset=CC3M2024.11 | — | 20.15 | — | — | — | — | |
| NegCLIPPre-training Dataset=CC3M2024.11 | — | 23.09 | — | — | — | — | |
| NegCLIPPre-training Dataset=CC12M2024.11 | — | 25.51 | — | — | — | — | |
| NegCLIP++Pre-training Dataset=CC3M2024.11 | — | 23.2 | — | — | — | — | |
| NegCLIP++Pre-training Dataset=CC12M2024.11 | — | 27.48 | — | — | — | — | |
| TripletCLIPPre-training Dataset=CC3M2024.11 | — | 25.11 | — | — | — | — | |
| TripletCLIPPre-training Dataset=CC12M2024.11 | — | 28.06 | — | — | — | — | |
| TripletCLIP++Pre-training Dataset=CC3M2024.11 | — | 26.64 | — | — | — | — |