Text-to-Image Retrieval on MS-COCO 5K (test)
68.3R@1BLIP-2 ViT-g
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| BLIP-2 ViT-gModel Category=Dual encoder + Fusion encoder reranking, #Trainable Params=1.2B, Evaluation Protocol=Fine-tuned2023.01 | 68.3 | — | 87.7 | 92.6 | — | — | |
| BEIT-3Model Category=Dual-encoder models, #Trainable Params=1.9B, Evaluation Protocol=Fine-tuned2023.01 | 67.2 | — | 87.7 | 92.8 | — | — | |
| BLIP-2 ViT-LModel Category=Dual encoder + Fusion encoder reranking, #Trainable Params=474M, Evaluation Protocol=Fine-tuned2023.01 | 66.3 | — | 86.5 | 91.8 | — | — | |
| ONE-PEACEEvaluation Setting=Fine-tuning2023.05 | 65.4 | — | 86.3 | 91.9 | — | — | |
| BEIT-3Evaluation Setting=Fine-tuning2023.05 | 65.1 | — | 86.6 | 92.3 | — | — | |
| BLIPModel Category=Dual encoder + Fusion encoder reranking, #Trainable Params=446M, Evaluation Protocol=Fine-tuned2023.01 | 65.1 | — | 86.3 | 91.8 | — | — | |
| OmniVLEvaluation Setting=Fine-tuning2023.05 | 64.8 | — | 86.1 | 91.6 | — | — | |
| UncompressedReduce Ratio=None, GFLOPS=153.2, Base Model=BLIP2024.03 | 64.3 | — | 85.7 | 91.5 | — | — | |
| BiCro*Noise=20%, mismatch_threshold=true2023.03 | 63.7 | — | 90.3 | 95.7 | — | — | |
| X-VLM oursPre-train=true, Data size=4M2022.10 | 63.6 | — | 86 | 91.5 | — | — | |
| FlorenceEvaluation Setting=Fine-tuning2023.05 | 63.2 | — | 85.7 | — | — | — | |
| FlorenceModel Category=Dual-encoder models, #Trainable Params=893M, Evaluation Protocol=Fine-tuned2023.01 | 63.2 | — | 85.7 | — | — | — | |
| X-VLMPre-train=true, Data size=4M2022.10 | 63.1 | — | 85.7 | 91.6 | — | — | |
| BLIP#PT Data=14M2022.12 | 63.1 | — | 85.3 | 91.1 | — | — | |
| BLIP2026.03 | 63.1 | — | 85.3 | — | — | — | |
| FIBER + EQSIMFine-tuning=COCO, Resolution=288x2882023.03 | 62.55 | — | 85.36 | 91.35 | — | — | |
| BiCroNoise=20%2023.03 | 62.5 | — | 89.8 | 95.5 | — | — | |
| BiCro*Noise=40%, mismatch_threshold=true2023.03 | 61.8 | — | 89.2 | 94.9 | — | — | |
| DECLNoise=20%2023.03 | 61.7 | — | 89.3 | 95.4 | — | — | |
| BiCroNoise=40%2023.03 | 61.5 | — | 89.4 | 95.5 | — | — | |
| FILIPEvaluation Setting=Fine-tuning2023.05 | 61.2 | — | 84.3 | 90.6 | — | — | |
| FILIPModel Category=Dual-encoder models, #Trainable Params=417M, Evaluation Protocol=Fine-tuned2023.01 | 61.2 | — | 84.3 | 90.6 | — | — | |
| NCRNoise=20%2023.03 | 60.8 | — | 88.8 | 95 | — | — | |
| ALBEFPre-train=true, Data size=14M2022.10 | 60.7 | — | 84.3 | 90.5 | — | — | |
| ALBEFModel Category=Dual encoder + Fusion encoder reranking, #Trainable Params=233M, Evaluation Protocol=Fine-tuned2023.01 | 60.7 | — | 84.3 | 90.5 | — | — | |
| ALBEF2026.03 | 60.7 | — | 84.3 | — | — | — | |
| MADTPReduce Ratio=0.5, GFLOPS=87.4, Base Model=BLIP2024.03 | 60.3 | — | 83.6 | 89.9 | — | — | |
| ALIGN#PT Data=1.2B2022.12 | 59.9 | — | 83.3 | 89.8 | — | — | |
| ALIGNEvaluation Setting=Fine-tuning2023.05 | 59.9 | — | 83.3 | 89.8 | — | — | |
| ALIGNModel Category=Dual-encoder models, #Trainable Params=820M, Evaluation Protocol=Fine-tuned2023.01 | 59.9 | — | 83.3 | 89.8 | — | — | |
| ALIGNPretrain=1.8B, GPUs=1024 TPUv3, Bs=16384, Zero-shot=false2024.04 | 59.9 | — | 83.3 | — | — | — | |
| UPopReduce Ratio=0.5, GFLOPS=88.3, Base Model=BLIP2024.03 | 59.8 | — | 83.1 | 89.8 | — | — | |
| NCRNoise=40%2023.03 | 59.6 | — | 88.1 | 94.7 | — | — | |
| DECLNoise=40%2023.03 | 59.5 | — | 88.3 | 94.8 | — | — | |
| FIBERFine-tuning=COCO, Resolution=288x2882023.03 | 59.31 | — | 83.73 | 90.43 | — | — | |
| VinVLPre-train=true, Data size=5.6M2022.10 | 58.8 | — | 83.5 | 90.3 | — | — | |
| VinVLModel Category=Fusion-encoder models, #Trainable Params=345M, Evaluation Protocol=Fine-tuned2023.01 | 58.8 | — | 83.5 | 90.3 | — | — | |
| BiCroNoise=60%2023.03 | 58.7 | — | 87 | 93.8 | — | — | |
| InternVL-Gmulti-lingual=true, Zero-shot=true2023.12 | 58.6 | — | 81.3 | 88 | — | 88.8 | |
| BiCro*Noise=60%, mismatch_threshold=true2023.03 | 58.3 | — | 87.2 | 93.9 | — | — | |
| SAFNoise=20%2023.03 | 57.8 | — | 86.4 | 91.9 | — | — | |
| CDDSBackbone=ViT-224-Large + BERT-Large, Patch size=16×162026.03 | 57.6 | — | 81.6 | — | — | — | |
| LAPSBackbone=ViT-224-Large + BERT-Large, Patch size=16×162026.03 | 57.1 | — | 81.3 | — | — | — | |
| DECLNoise=60%2023.03 | 57 | — | 86.6 | 93.8 | — | — | |
| ALBEF#PT Data=4M2022.12 | 56.8 | — | 81.5 | 89.2 | — | — | |
| ALBEFPretrain=4.0M, GPUs=8 A100, Bs=5122024.04 | 56.8 | — | 81.5 | — | — | — | |
| VSE++Backbone=ViT-224-Large + BERT-Large, Patch size=16×162026.03 | 56.7 | — | 81.9 | — | — | — | |
| IMRAMNoise=20%2023.03 | 55.9 | — | 84.4 | 89.6 | — | — | |
| HiTeA#PT Data=5M2022.12 | 55.6 | — | 80.6 | 87.8 | — | — | |
| FIBER2023.03 | 55.19 | — | 81.49 | 88.89 | — | — | |
| CDDSBackbone=ViT-224 + BERT, Patch size=14×142026.03 | 54.8 | — | 80.9 | — | — | — | |
| Singularity#PT Data=5M2022.12 | 54.6 | — | 80 | 87.8 | — | — | |
| LAPSBackbone=ViT-224 + BERT, Patch size=14×142026.03 | 54.3 | — | 80 | — | — | — | |
| MaMMUTimage model size=630M, evaluation protocol=zero-shot, model type=dual-encoder2023.03 | 54.1 | — | 76.8 | 84.2 | — | — | |
| InternVL-Cmulti-lingual=true, Zero-shot=true2023.12 | 54.1 | — | 77.3 | 84.6 | — | 86.6 | |
| OSCAR#PT Data=4M2022.12 | 54 | — | 80.8 | 88.5 | — | — | |
| OSCARModel Category=Fusion-encoder models, #Trainable Params=345M, Evaluation Protocol=Fine-tuned2023.01 | 54 | — | 80.8 | 88.5 | — | — | |
| OSCARPretrain=6.5M, GPUs=16 V100, Bs=1,0242024.04 | 54 | — | 80.8 | — | — | — | |
| VSE∞(WSL) oursPre-train=false2022.10 | 53.7 | — | 80.6 | 88.4 | — | — | |
| VSE++Backbone=ViT-224 + BERT, Patch size=14×142026.03 | 53.6 | — | 79.7 | — | — | — | |
| Ours†Backbone=ResNeXt-101 + BERT, CA (Cross-Attention)=false, Ensemble=true2022.11 | 53.4 | — | 80.9 | 88.6 | 482 | — | |
| MADTPReduce Ratio=0.75, GFLOPS=50.2, Base Model=BLIP2024.03 | 53.4 | — | 78.4 | 86.2 | — | — | |
| SCANBackbone=ViT-224-Large + BERT-Large, Patch size=16×162026.03 | 53.2 | — | 80.7 | — | — | — | |
| UNITER#PT Data=4M2022.12 | 52.9 | — | 79.9 | 88 | — | — | |
| UNITERModel Category=Fusion-encoder models, #Trainable Params=303M, Evaluation Protocol=Fine-tuned2023.01 | 52.9 | — | 79.9 | 88 | — | — | |
| VSE∞†Backbone=ResNeXt-101 + BERT, CA (Cross-Attention)=false, Ensemble=true2022.11 | 52.7 | — | 80.2 | 88.3 | 474.8 | — | |
| OursBackbone=ResNeXt-101 + BERT, CA (Cross-Attention)=false, Ensemble=false2022.11 | 52.1 | — | 79.6 | 87.8 | 474.9 | — | |
| RO-ViTimage backbone size=303M, zero-shot=true2023.05 | 51.8 | — | 75 | 83 | — | — | |
| VSE∞(WSL)Pre-train=false2022.10 | 51.6 | — | 79.3 | — | — | — | |
| VSE∞Backbone=ResNeXt-101 + BERT, CA (Cross-Attention)=false, Ensemble=false2022.11 | 51.6 | — | 79.3 | 87.6 | 468.9 | — | |
| OpenCLIP-Gmulti-lingual=false, Zero-shot=true2023.12 | 51.4 | — | 74.9 | 83 | — | 85 | |
| CoCaEvaluation Setting=Zero-shot2023.05 | 51.2 | — | 74.2 | 82 | — | — | |
| CoCaimage model size=1B, evaluation protocol=zero-shot, model type=dual-encoder2023.03 | 51.2 | — | 74.2 | 82 | — | — | |
| CoCaimage backbone size=1B, zero-shot=true2023.05 | 51.2 | — | 74.2 | 82 | — | — | |
| CoCamulti-lingual=false, Zero-shot=true2023.12 | 51.2 | — | 74.2 | 82 | — | 84.8 | |
| EVA-02-CLIP-E+multi-lingual=false, Zero-shot=true2023.12 | 51.1 | — | 75 | 82.7 | — | 85.1 | |
| SCANBackbone=ViT-224 + BERT, Patch size=14×142026.03 | 50.7 | — | 77.6 | — | — | — | |
| SOHO2026.03 | 50.6 | — | 78 | — | — | — | |
| EVA-01-CLIP-g+multi-lingual=false, Zero-shot=true2023.12 | 50.3 | — | 74 | 82.1 | — | 84.6 | |
| 3SHNetType=Region+Grid, Ensemble=true2024.04 | 50.3 | — | 79.3 | 87.7 | 471.2 | — | |
| 3SHNetGPUs=1 TitanX, Bs=2562024.04 | 50.3 | — | 79.3 | — | — | — | |
| CoCa-Limage model size=303M, evaluation protocol=zero-shot, model type=dual-encoder2023.03 | 50.1 | — | 73.8 | 81.8 | — | — | |
| CoCa-Largeimage backbone size=303M, zero-shot=true2023.05 | 50.1 | — | 73.8 | 81.8 | — | — | |
| CoCa-Largeimage encoder size=303M, Zero-shot=true2023.09 | 50.1 | — | 73.8 | 81.8 | — | — | |
| VSRNNoise=20%2023.03 | 50 | — | 80.3 | 88.3 | — | — | |
| 3SHNetType=Region+Grid, Ensemble=false2024.04 | 49.9 | — | 78.8 | 87.2 | 468 | — | |
| CFM-ViTimage encoder size=303M, Zero-shot=true2023.09 | 49.8 | — | 73.5 | 81.6 | — | — | |
| OpenCLIP-Hmulti-lingual=false, Zero-shot=true2023.12 | 49.5 | — | 73.4 | 81.5 | — | 83.9 | |
| OpenCLIP-XLM-R-Hmulti-lingual=true, Zero-shot=true2023.12 | 49.3 | — | 73.2 | 81.5 | — | 84 | |
| 3SHNetType=Grid, Ensemble=true2024.04 | 49 | — | 78.3 | 86.8 | 464.8 | — | |
| OpenCLIP-gmulti-lingual=false, Zero-shot=true2023.12 | 48.8 | — | 73.3 | 81.5 | — | 83.9 | |
| UNITERPretrain=9.6M, GPUs=16 V100, Bs=1922024.04 | 48.4 | — | 76.7 | — | — | — | |
| VSE∞Backbone=ViT-B/322023.09 | 48.2 | — | 76.7 | 85.5 | — | — | |
| 3SHNetType=Region, Ensemble=true2024.04 | 48.2 | — | 77.5 | 86.3 | 460.2 | — | |
| PCME++Backbone=ViT-B/322023.09 | 48.1 | — | 76.7 | 85.5 | — | — | |
| ONE-PEACEEvaluation Setting=Zero-shot2023.05 | 48 | — | 71.5 | 79.6 | — | — | |
| ONE-PEACEmulti-lingual=false, Zero-shot=true2023.12 | 48 | — | 71.5 | — | — | 83.2 | |
| 3SHNetType=Grid, Ensemble=false2024.04 | 48 | — | 77.4 | 86.3 | 459 | — | |
| VSE∞ w/ Seg.Type=Region+Grid, Ensemble=false2024.04 | 47.8 | — | 76.9 | 86 | 457.1 | — | |
| VISTA-LTime (ms)=~40, Zero-shot=true2022.03 | 47.4 | — | 75 | 84 | — | — |