Multimodal Compositional Understanding on ARO
86.7Relational ScoreCapPa
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CapPamode=Zero-shot2023.06 | 86.7 | 85.7 | — | 99.2 | 98.8 | |
| Capmode=Zero-shot2023.06 | 86.6 | 88.9 | — | 99.1 | 99 | |
| Blind dec.mode=Zero-shot2023.06 | 86.2 | 83.7 | — | 98.8 | 98.7 | |
| MosaiCLIPArch.=Swin-T, Pre-training data=CC-12M2023.05 | 84.3 | 76.8 | 55.5 | — | — | |
| MosaiCLIP2023.05 | 83.7 | 78 | 79.4 | — | — | |
| MosaiCLIP NoCurriccurriculum_learning=false2023.05 | 83.2 | 78.6 | 77.3 | — | — | |
| MosaiCLIP WiSE-FTfine_tuning=WiSE-FT2023.05 | 82.6 | 77 | 79.9 | — | — | |
| NegCLIPArch.=Swin-T, Pre-training data=CC-12M2023.05 | 82.4 | 66.8 | 59.7 | — | — | |
| MosaiCLIPArch.=RN-50, Pre-training data=CC-12M2023.05 | 82 | 78.5 | 55.4 | — | — | |
| NegCLIPmode=Zero-shot2023.06 | 81 | 71 | — | 91 | 86 | |
| NegCLIPArch.=RN-50, Pre-training data=CC-12M2023.05 | 80.5 | 66.5 | 60.5 | — | — | |
| MosaiCLIPArch.=RN-50, Pre-training data=YFCC-15M2023.05 | 76.3 | 68.9 | 38.2 | — | — | |
| MosaiCLIPArch.=Swin-T, Pre-training data=YFCC-15M2023.05 | 74.7 | 66.1 | 35.8 | — | — | |
| NegCLIPArch.=Swin-T, Pre-training data=YFCC-15M2023.05 | 73.6 | 58.9 | 35.5 | — | — | |
| ARO Bestmode=Zero-shot2023.06 | 73 | 88 | — | 60 | 46 | |
| Syn-CLIPtraining_data=Million-scale synthetic data2023.05 | 71.4 | 66.9 | 65.1 | — | — | |
| NegCLIPArch.=RN-50, Pre-training data=YFCC-15M2023.05 | 68 | 58.5 | 37.1 | — | — | |
| CLIPmode=Zero-shot2023.05 | 59.9 | 63.6 | 53.3 | — | — | |
| CLIPmode=Zero-shot2023.06 | 58.7 | 62.7 | — | 57.9 | 49.5 | |
| CLIPArch.=RN-50, Pre-training data=YFCC-15M2023.05 | 57.8 | 55.1 | 18.3 | — | — | |
| CLIPArch.=Swin-T, Pre-training data=YFCC-15M2023.05 | 53.8 | 56.2 | 18.4 | — | — | |
| CLIPArch.=RN-50, Pre-training data=CC-12M2023.05 | 52.9 | 59.7 | 22.6 | — | — | |
| CLIPArch.=Swin-T, Pre-training data=CC-12M2023.05 | 51 | 56.6 | 25.5 | — | — | |
| RandomPre-training data=CC-12M2023.05 | 50 | 50 | 20 | — | — | |
| RandomPre-training data=YFCC-15M2023.05 | 50 | 50 | 20 | — | — | |
| CLIP-FTmode=Fine-tuned2023.05 | 41.7 | 59.3 | 25.2 | — | — | |
| CLIP*mode=Zero-shot2023.06 | 39.7 | 53.2 | — | 45.5 | 37 |