Image-to-text retrieval on SugarCrepe (Compositional R@1)
98.44R@1 (Add)JAM
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| JAMLanguage Backbone=OLMo2 (7B), Vision Backbone=ResNet50, Alignment Method=Spread2025.07 | 98.44 | 86.9 | 80.46 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Llama3.2 (1B), Vision Backbone=DINOv2 (ViT-B), Alignment Method=Spread2025.07 | 97.83 | 87.21 | 79.05 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=OLMo2 (7B), Vision Backbone=DINOv2 (ViT-B), Alignment Method=Spread2025.07 | 97.41 | 88.32 | 77.62 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Gemma2 (2B), Vision Backbone=DINOv2 (ViT-B; 86M), Alignment Method=NegCon2025.07 | 96.85 | 86.32 | 74.5 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Llama3.2 (1B), Vision Backbone=ResNet50, Alignment Method=Spread2025.07 | 96.85 | 87.17 | 82.17 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=OLMo2 (7B), Vision Backbone=DINOv2 (ViT-B), Alignment Method=NegCon2025.07 | 96.67 | 85.3 | 71.69 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Gemma2 (2B), Vision Backbone=DINOv2 (ViT-B; 86M), Alignment Method=Spread2025.07 | 95.74 | 88.01 | 80.2 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Gemma2 (2B), Vision Backbone=ResNet50, Alignment Method=Spread2025.07 | 94.81 | 87.6 | 82.17 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Gemma2 (2B), Vision Backbone=ResNet50, Alignment Method=NegCon2025.07 | 94.8 | 87.28 | 72.27 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Llama3.2 (1B), Vision Backbone=DINOv2 (ViT-B), Alignment Method=NegCon2025.07 | 94.25 | 83.43 | 68.26 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Llama3.2 (1B), Vision Backbone=ResNet50, Alignment Method=NegCon2025.07 | 94.1 | 85.74 | 71.69 | — | — | — | — | — | — | — | — | — | — | |
| Finetuned CLIP (ViT-B/32)Backbone=ViT-B/32, mode=fine-tuned2025.07 | 92.69 | 74.62 | 67.21 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=OLMo2 (7B), Vision Backbone=ResNet50, Alignment Method=NegCon2025.07 | 90.12 | 89.69 | 71.38 | — | — | — | — | — | — | — | — | — | — | |
| Pretrained CLIP (ViT-B-32/SigLIP)Backbone=ViT-B-32, Training Data=SigLIP2025.07 | 86.56 | 85.01 | 70.97 | — | — | — | — | — | — | — | — | — | — | |
| Pretrained CLIP (ViT-B-32/LAION-400m)Backbone=ViT-B-32, Training Data=LAION-400m2025.07 | 79.64 | 80.9 | 67.3 | — | — | — | — | — | — | — | — | — | — | |
| Pretrained CLIP (ViT-B-32/OpenAI)Backbone=ViT-B-32, Training Data=OpenAI2025.07 | 77.58 | 81.05 | 64.69 | — | — | — | — | — | — | — | — | — | — | |
| IL-CLIPPre-training Dataset=CC12M2024.04 | 73.8 | 73 | 62.9 | — | — | — | — | — | — | — | — | — | — | |
| Codebook-CLIPPre-training Dataset=CC12M2024.04 | 71.3 | 71.1 | 59.5 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Llama3.2 (1B), Vision Backbone=ResNet50, Alignment Method=Con2025.07 | 67.94 | 67.36 | 71.11 | — | — | — | — | — | — | — | — | — | — | |
| CLIPPre-training Dataset=CC12M2024.04 | 67.5 | 70 | 60.2 | — | — | — | — | — | — | — | — | — | — | |
| IL-CLIPPre-training Dataset=CC3M2024.04 | 66.1 | 67 | 54.5 | — | — | — | — | — | — | — | — | — | — | |
| Codebook-CLIPPre-training Dataset=CC3M2024.04 | 65.9 | 64.8 | 54.9 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=OLMo2 (7B), Vision Backbone=DINOv2 (ViT-B), Alignment Method=Con2025.07 | 65.17 | 68.48 | 66.88 | — | — | — | — | — | — | — | — | — | — | |
| NegCLIPPre-training Dataset=CC12M2024.04 | 65 | 70.2 | 67.2 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=OLMo2 (7B), Vision Backbone=ResNet50, Alignment Method=Con2025.07 | 64.35 | 68.01 | 73.38 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Gemma2 (2B), Vision Backbone=ResNet50, Alignment Method=Con2025.07 | 63.02 | 74.23 | 63.75 | — | — | — | — | — | — | — | — | — | — | |
| CLIPPre-training Dataset=CC3M2024.04 | 61.9 | 64.3 | 52.9 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Llama3.2 (1B), Vision Backbone=DINOv2 (ViT-B), Alignment Method=Con2025.07 | 60.99 | 66.97 | 66.59 | — | — | — | — | — | — | — | — | — | — | |
| NegCLIPPre-training Dataset=CC3M2024.04 | 59.3 | 59.2 | 60.1 | — | — | — | — | — | — | — | — | — | — | |
| JAMLanguage Backbone=Gemma2 (2B), Vision Backbone=DINOv2 (ViT-B; 86M), Alignment Method=Con2025.07 | 58.34 | 65.14 | 57.36 | — | — | — | — | — | — | — | — | — | — | |
| CLIPPre-training Dataset=LAION2B, Source=Reproduced2024.12 | — | — | — | 93.77 | 82.49 | 68.92 | 60 | 67.42 | 87.1 | 77.89 | 81.73 | 63.71 | 82.5 | |
| Neg-CLIPPre-training Dataset=LAION2B, Source=Reproduced2024.12 | — | — | — | 92.62 | 84.64 | 71.41 | 74.69 | 76.58 | 86.71 | 85.26 | 82.89 | 75.64 | 85.99 | |
| Neg-CLIPPre-training Dataset=CC12M, Source=Original Paper2024.12 | — | — | — | 77.84 | 69.29 | 63.23 | 66.53 | 62.31 | 67.71 | 69.65 | 70.12 | 64.42 | 68.41 | |
| Structure-CLIPPre-training Dataset=LAION2B, Source=Reproduced2024.12 | — | — | — | 91.77 | 86.8 | 75.32 | 74.29 | 84.83 | 88.94 | 88.01 | 84.63 | 79.56 | 88.48 | |
| TripletCLIPPre-training Dataset=CC12M, Source=Original Paper2024.12 | — | — | — | 83.66 | 81.22 | 79.02 | 64.49 | 63.66 | 73.67 | 75.43 | 81.3 | 64.08 | 74.55 | |
| YUKINOPre-training Dataset=LAION2B, Source=Reproduced2024.12 | — | — | — | 97.15 | 94.8 | 91.25 | 88.57 | 89.79 | 96.65 | 93.21 | 94.4 | 89.18 | 94.93 | |
| YUKINO-OTIPre-training Dataset=LAION2B, Source=Reproduced2024.12 | — | — | — | 97.82 | 95.43 | 91.68 | 87.76 | 90.69 | 96.51 | 94.8 | 94.98 | 89.23 | 95.66 |