Compositional Reasoning on SugarCrepe
95.2Overall AccuracyInternVL2-8B
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| InternVL2-8BModel=InternVL2-8B2026.06 | 95.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama-90BModel=Llama-90B2026.06 | 94.71 | — | — | — | — | — | — | — | — | — | — | — | — | |
| OneVision-4BModel=OneVision-4B2026.06 | 94.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| OneVision-8BModel=OneVision-8B2026.06 | 94.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| OneVision-4B-LNModel=OneVision-4B-LN2026.06 | 89.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonCLIP + READViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true2026.06 | 88.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TinyLLaVALLM=Qwen2.5, Size=3B2026.07 | 88.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-13BModel=LLaVA-13B2026.06 | 87.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TinyLLaVALLM=Qwen2.5, Size=1.5B2026.07 | 87.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CE-CLIP+Evaluation Protocol=Zero-shot2023.06 | 87.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CE-CLIP+Training Source=COCO+CC3M, Num Synthetic Captions=15M2025.03 | 87.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TinyLLaVALLM=Qwen1.5, Size=4B2026.07 | 87.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRAGLLM=Qwen2.5, Size=1.5B2026.07 | 87.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPARCLTraining Source=COCO, Num Synthetic Images=820K, Num Synthetic Captions=820K2025.03 | 87.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| READ-CLIPViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true2026.06 | 87 | — | — | — | — | — | — | — | — | — | — | — | — | |
| COSMOSData Size=30M2024.12 | 86.6 | — | — | — | 87.3 | — | — | 79.1 | — | — | 88.1 | — | — | |
| Cambrian-8BModel=Cambrian-8B2026.06 | 86.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DACEvaluation Protocol=finetuned2024.11 | 86.41 | 94.43 | 89.48 | 84.35 | — | 75.1 | 74.17 | — | 89.67 | 97.69 | — | — | — | |
| CE-CLIPViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true2026.06 | 86 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CE-CLIPEvaluation Protocol=Zero-shot2023.06 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| AMR-NegCLIPTraining Source=COCO, Num Synthetic Captions=500K2025.03 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CE-CLIPTraining Source=COCO, Num Synthetic Captions=2M2025.03 | 85.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| FSC-CLIPTraining Source=COCO, Num Synthetic Captions=1.5M2025.03 | 85.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CLOVETraining Source=LAION-COCO, Num Synthetic Captions=>1B2025.03 | 85.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonSigLIPViT=So/14, Resolution=384, Data=+58M2026.06 | 84.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| NegCLIPViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true2026.06 | 83.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenCLIPData Size=2B2024.12 | 83.5 | — | — | — | 84.7 | — | — | 67 | — | — | 87.4 | — | — | |
| TRAGLLM=Qwen1.5, Size=1.8B2026.07 | 83.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SigLIPViT=So/14, Resolution=384, Data=10.0B2026.06 | 83 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SigLIP2ViT=G/16, Resolution=384, Data=10.0B2026.06 | 83 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Long-CLIPViT=L/14, Resolution=224, Data=0.4B2026.06 | 82.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CounterCurateTraining Source=Flickr, Num Synthetic Images=150K, Num Synthetic Captions=150K2025.03 | 82.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Triplet-CLIPViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true2026.06 | 82.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| NegCLIPTraining Source=COCO, Num Synthetic Captions=500K2025.03 | 82.5 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripletCLIPEvaluation Protocol=finetuned2024.11 | 82.46 | 94.43 | 85.53 | 80.94 | — | 69.8 | 69.82 | — | 90.4 | 86.27 | — | — | — | |
| OpenCLIPData Size=1B2024.12 | 81.9 | — | — | — | 82.7 | — | — | 64.6 | — | — | 86.6 | — | — | |
| DreamLIPData Size=30M2024.12 | 81.8 | — | — | — | 82.7 | — | — | 76.5 | — | — | 82.1 | — | — | |
| LLaVA-7BModel=LLaVA-7B2026.06 | 81.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CompoDistillLLM=Qwen1.5, Size=1.8B2026.07 | 81.7 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Eva-CLIP-02ViT=L/14, Resolution=224, Data=2.0B2026.06 | 81.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SigLIP2ViT=L/16, Resolution=384, Data=10.0B2026.06 | 81.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MetaCLIPViT=L/14, Resolution=224, Data=2.5B2026.06 | 80.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| NegCLIPEvaluation Protocol=finetuned2024.11 | 80.59 | 91.53 | 83.25 | 73.97 | — | 72.24 | 67.72 | — | 86.95 | 88.44 | — | — | — | |
| OpenCLIPData Size=400M2024.12 | 80 | — | — | — | 81.6 | — | — | 65.2 | — | — | 82.5 | — | — | |
| CLIPTraining Source=COCO, Training Protocol=Finetune2025.03 | 79.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRAGLLM=Qwen2.5, Size=0.5B2026.07 | 78.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonCLIPViT=L/14, Resolution=224, Data=+58M2026.06 | 78.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonCLIPViT=L/14, Resolution=336, Data=+58M2026.06 | 78.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GNM-CLIPViT=B/32, Resolution=224, Fine-tuned on MS-COCO=true2026.06 | 77.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BaselineEvaluation Protocol=finetuned2024.11 | 77.84 | 93.22 | 84.39 | 67.35 | — | 62.04 | 70.12 | — | 88.31 | 79.48 | — | — | — | |
| TSVLC (RB)Evaluation Protocol=finetuned2024.11 | 77.84 | 91.34 | 81.34 | 64.15 | — | 68.16 | 69.07 | — | 79.49 | 91.33 | — | — | — | |
| CLIP-FTEvaluation Protocol=Fine-tuned on COCO2023.06 | 77.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| MetaCLIPViT=B/32, Resolution=224, Data=0.4B2026.06 | 76.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| ReasonSigLIP2ViT=G/16, Resolution=384, Data=+58M2026.06 | 76.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-KDLLM=Qwen1.5, Size=1.8B2026.07 | 76.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CLIPData Size=30M2024.12 | 76.5 | — | — | — | 76.2 | — | — | 68.5 | — | — | 79.5 | — | — | |
| KDLLM=Qwen1.5, Size=1.8B2026.07 | 76.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SigLIPData Size=30M2024.12 | 76 | — | — | — | 76.6 | — | — | 67.8 | — | — | 77.9 | — | — | |
| ReasonCLIPViT=B/32, Resolution=224, Data=+58M2026.06 | 75.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CON-CLIPEvaluation Protocol=finetuned2024.11 | 75.58 | 93.58 | 80.96 | 63.3 | — | 87.29 | 79.62 | — | 59.18 | 65.16 | — | — | — | |
| CLIPS (ViT-L/14)Backbone=ViT-L/142026.03 | 75.2 | — | — | — | 86.1 | — | — | 77.8 | — | — | 86.6 | — | — | |
| TinyLLaVALLM=Qwen2.5, Size=0.5B2026.07 | 75.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| C^2LIPBackbone=ViT-B/162026.03 | 75 | — | — | — | 88.3 | — | — | 73.1 | — | — | 94.2 | — | — | |
| CLIPViT=L/14, Resolution=336, Data=0.4B2026.06 | 74.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TinyLLaVALLM=Qwen1.5, Size=1.8B2026.07 | 74.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CLIPEvaluation Protocol=finetuned2024.11 | 74.73 | 90.92 | 79.69 | 64.01 | — | 60.82 | 64.26 | — | 84.67 | 78.76 | — | — | — | |
| FIGCLIPTraining Source=VidSitu, Num Real Images=20K videos2025.03 | 74.6 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripletCLIPPre-training Dataset=CC12M2024.11 | 74.55 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SigLIP2-ViT-B/16Backbone=ViT-B/162026.03 | 74.4 | — | — | — | 86 | — | — | 71.9 | — | — | 89 | — | — | |
| TSVLC (LLM+RB)Evaluation Protocol=finetuned2024.11 | 73.61 | 88.13 | 76.78 | 62.73 | — | 64.08 | 66.67 | — | 75.8 | 81.07 | — | — | — | |
| CLIPTraining Protocol=Zero-Shot2025.03 | 73.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| SigLIP ViT-L/16-res256Backbone=ViT-L/16, Input Resolution=256x2562026.03 | 73.4 | — | — | — | 84 | — | — | 72.8 | — | — | 86.7 | — | — | |
| CLIPViT=L/14, Resolution=224, Data=0.4B2026.06 | 73.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TRAGLLM=Qwen1.5, Size=0.5B2026.07 | 73.4 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CLIPEvaluation Protocol=Zero-shot2023.06 | 73.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CLIPViT=B/32, Resolution=224, Data=0.4B2026.06 | 73.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CLIPEvaluation Protocol=Standard2024.11 | 73.06 | 90.92 | 80.08 | 69.13 | — | 61.22 | 64.26 | — | 77.16 | 68.64 | — | — | — | |
| NegCLIP++Pre-training Dataset=CC12M2024.11 | 73.05 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CompoDistillLLM=Qwen1.5, Size=0.5B2026.07 | 72.1 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP-BBackbone=ViT-B2026.03 | 71.3 | — | — | — | 84.1 | — | — | 73.6 | — | — | 92.2 | — | — | |
| SF-CLIPTraining Source=YFCC15M2025.03 | 71.2 | — | — | — | — | — | — | — | — | — | — | — | — | |
| CLIP-A (ViT-L/14)Backbone=ViT-L/142026.03 | 70.5 | — | — | — | 82.9 | — | — | 63.8 | — | — | 85.8 | — | — | |
| IL-CLIPTraining Source=CC12M2025.03 | 70.3 | — | — | — | — | — | — | — | — | — | — | — | — | |
| FLAVABackbone=ViT-B2026.03 | 69.9 | — | — | — | 84.9 | — | — | 73.8 | — | — | 79.6 | — | — | |
| NegCLIPPre-training Dataset=CC12M2024.11 | 68.41 | — | — | — | — | — | — | — | — | — | — | — | — | |
| LaCLIPPre-training Dataset=CC12M2024.11 | 67.21 | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-KDLLM=Qwen1.5, Size=0.5B2026.07 | 66.9 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripletCLIP++Pre-training Dataset=CC3M2024.11 | 66.09 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TripletCLIPPre-training Dataset=CC3M2024.11 | 63.49 | — | — | — | — | — | — | — | — | — | — | — | — | |
| NegCLIP++Pre-training Dataset=CC3M2024.11 | 61.69 | — | — | — | — | — | — | — | — | — | — | — | — | |
| NegCLIPPre-training Dataset=CC3M2024.11 | 57.18 | — | — | — | — | — | — | — | — | — | — | — | — | |
| LaCLIPPre-training Dataset=CC3M2024.11 | 54.09 | — | — | — | — | — | — | — | — | — | — | — | — | |
| LaCLIP + HNPre-training Dataset=CC3M2024.11 | 53.92 | — | — | — | — | — | — | — | — | — | — | — | — | |
| TinyLLaVALLM=Qwen1.5, Size=0.5B2026.07 | 52.8 | — | — | — | — | — | — | — | — | — | — | — | — | |
| KDLLM=Qwen1.5, Size=0.5B2026.07 | 51 | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP22023.06 | — | — | — | — | 0.867 | — | — | 0.698 | — | — | 0.865 | — | — | |
| CE-CLIP2023.06 | — | 0.931 | 0.888 | 0.79 | 0.87 | 0.728 | 0.77 | 0.749 | 0.924 | 0.934 | 0.929 | — | — | |
| CE-CLIP+2023.06 | — | 0.938 | 0.908 | 0.832 | 0.893 | 0.768 | 0.793 | 0.78 | 0.938 | 0.949 | 0.944 | — | — | |
| CLIP2023.06 | — | 0.909 | 0.8 | 0.692 | 0.802 | 0.614 | 0.64 | 0.627 | 0.772 | 0.682 | 0.727 | — | — | |
| CLIPBackbone=CLIP ViT-B/162026.06 | — | — | — | — | — | — | — | — | — | — | — | 66.3 | 70.5 |