Compositional Evaluation on SugarCrepe (test)
95.52Replace (Object)DataComp (xlarge)
Evaluation results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DataComp (xlarge)Backbone=ViT-L/14+, Samples Seen=13B2024.11 | 95.52 | — | — | — | 84.52 | 69.99 | 65.04 | 66.82 | 91.03 | 84.97 | — | — | — | 79.7 | |
| DataComp (large)Backbone=ViT-B/16, Samples Seen=1B2024.11 | 92.68 | — | — | — | 79.82 | 63.94 | 56.1 | 57.66 | 84.34 | 78.61 | — | — | — | 73.31 | |
| Cluster Masking (RGB0.5)Clustering Method=RGB, beta=0.52024.05 | 86.86 | — | — | — | 73.47 | 60.24 | 59.35 | 63.36 | 73.33 | 66.76 | 73.18 | 68.42 | 60.24 | — | |
| FLIP Attn-0.5Masking=Attention, Ratio=0.52024.05 | 86.62 | — | — | — | 75.5 | 63.22 | 52.44 | 63.06 | 71.65 | 66.76 | 71.57 | 68.39 | 63.22 | — | |
| Cluster Masking (RGB0.3)Clustering Method=RGB, beta=0.32024.05 | 86.13 | — | — | — | 75.13 | 64.65 | 66.67 | 63.36 | 74.92 | 71.24 | 75.91 | 69.91 | 64.65 | — | |
| FLIP Attn-0.3Masking=Attention, Ratio=0.32024.05 | 86.07 | — | — | — | 75 | 62.23 | 60.57 | 59.16 | 74.68 | 68.64 | 72.82 | 63.47 | 62.23 | — | |
| CLIP2024.05 | 85.77 | — | — | — | 79.18 | 64.51 | 61.78 | 58.71 | 74.24 | 68.35 | 73.71 | 68.75 | 64.51 | — | |
| Cluster Masking (K-means)Clustering Method=K-means2024.05 | 84.5 | — | — | — | 76.9 | 63.09 | 62.2 | 61.86 | 71.77 | 65.46 | 73.66 | 67.6 | 63.09 | — | |
| FLIP2024.05 | 84.07 | — | — | — | 75.88 | 66 | 60.16 | 61.56 | 71.67 | 63.15 | 71.97 | 66.86 | 66 | — | |
| TripletCLIPTraining Data=CC12M, Backbone=ViT-B/322024.11 | 83.66 | — | — | — | 81.22 | 79.02 | 64.49 | 63.66 | 73.67 | 75.43 | — | — | — | 74.45 | |
| NegCLIP++Training Data=CC12M, Backbone=ViT-B/322024.11 | 82.99 | — | — | — | 78.68 | 75.75 | 61.63 | 65.47 | 70.08 | 76.01 | — | — | — | 72.94 | |
| NegCLIPTraining Data=CC12M, Backbone=ViT-B/322024.11 | 77.84 | — | — | — | 69.29 | 63.23 | 66.53 | 62.31 | 67.17 | 69.65 | — | — | — | 68 | |
| DataComp (medium)Backbone=ViT-B/32, Samples Seen=128M2024.11 | 77 | — | — | — | 69.54 | 57.68 | 57.72 | 57.06 | 66.73 | 64.88 | — | — | — | 64.37 | |
| LaCLIPTraining Data=CC12M, Backbone=ViT-B/322024.11 | 75.06 | — | — | — | 65.48 | 58.68 | 53.47 | 57.66 | 67.65 | 66.76 | — | — | — | 63.54 | |
| TripletCLIPTraining Data=CC3M, Backbone=ViT-B/322024.11 | 69.92 | — | — | — | 69.03 | 64.72 | 56.33 | 57.96 | 62.61 | 63.87 | — | — | — | 63.49 | |
| NegCLIP++Training Data=CC3M, Backbone=ViT-B/322024.11 | 64.77 | — | — | — | 66.12 | 65.93 | 55.51 | 55.41 | 59.65 | 64.45 | — | — | — | 61.69 | |
| LaCLIP + HNTraining Data=CC3M, Backbone=ViT-B/322024.11 | 63.44 | — | — | — | 55.96 | 50.71 | 50.6 | 48.57 | 56.98 | 51.16 | — | — | — | 53.92 | |
| NegCLIPTraining Data=CC3M, Backbone=ViT-B/322024.11 | 62.71 | — | — | — | 58.12 | 54.48 | 56.33 | 51.2 | 56.26 | 61.13 | — | — | — | 57.18 | |
| LaCLIPTraining Data=CC3M, Backbone=ViT-B/322024.11 | 59.44 | — | — | — | 53.17 | 51.42 | 54.69 | 49.25 | 55.29 | 55.35 | — | — | — | 54.09 | |
| DataComp (small)Backbone=ViT-B/32, Samples Seen=13M2024.11 | 56.9 | — | — | — | 56.85 | 51.99 | 50.81 | 50 | 53.93 | 60.55 | — | — | — | 54.43 | |
| CLIPPre-training Dataset=DataComp-10M2024.04 | — | 63 | 62 | 57 | — | — | — | — | — | — | — | — | — | — | |
| CLIPTraining Dataset=-, Frozen=-2025.02 | — | 73 | 80 | 62 | — | — | — | — | — | — | — | — | — | — | |
| Codebook-CLIPPre-training Dataset=DataComp-10M2024.04 | — | 64 | 64 | 59 | — | — | — | — | — | — | — | — | — | — | |
| IL-CLIPPre-training Dataset=DataComp-10M2024.04 | — | 66 | 66 | 62 | — | — | — | — | — | — | — | — | — | — | |
| LABCLIPTraining Dataset=COCO, Frozen=true2025.02 | — | 81 | 82 | 74 | — | — | — | — | — | — | — | — | — | — | |
| LABCLIPTraining Dataset=CC3M, Frozen=true2025.02 | — | 83 | 80 | 68 | — | — | — | — | — | — | — | — | — | — | |
| NegCLIPPre-training Dataset=DataComp-10M2024.04 | — | 62 | 63 | 64 | — | — | — | — | — | — | — | — | — | — | |
| NegCLIPTraining Dataset=COCO, Frozen=false2025.02 | — | 87 | 85 | 75 | — | — | — | — | — | — | — | — | — | — |