Image-to-Text Retrieval on MS-COCO 5K (test)
85.4R@1BLIP-2 ViT-g
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| BLIP-2 ViT-gModel Category=Dual encoder + Fusion encoder reranking, #Trainable Params=1.2B, Evaluation Protocol=Fine-tuned2023.01 | 85.4 | 97 | 98.5 | — | — | |
| BEIT-3Architecture Type=Dual-encoder, Evaluation Protocol=Finetuning2022.08 | 84.8 | 96.5 | 98.3 | — | — | |
| BEiT-3Pre-training Images=35M+, Parameters=1.9B, Evaluation Protocol=Fine-tuned2022.12 | 84.8 | 96.5 | 98.3 | — | — | |
| BEIT-3Model Category=Dual-encoder models, #Trainable Params=1.9B, Evaluation Protocol=Fine-tuned2023.01 | 84.8 | 96.5 | 98.3 | — | — | |
| ONE-PEACEEvaluation Setting=Fine-tuning2023.05 | 84.1 | 96.3 | 98.3 | — | — | |
| BLIP-2 ViT-LModel Category=Dual encoder + Fusion encoder reranking, #Trainable Params=474M, Evaluation Protocol=Fine-tuned2023.01 | 83.5 | 96 | 98 | — | — | |
| mPLUGPretrain data=14M2022.05 | 82.8 | 96.1 | 98.3 | — | — | |
| BEIT-3Evaluation Setting=Fine-tuning2023.05 | 82.7 | 96 | 98.2 | — | — | |
| BLIP_ViT-LPre-train # Images=129M2022.01 | 82.4 | 95.4 | 97.9 | — | — | |
| BLIPArchitecture Type=Dual encoder + Fusion encoder reranking, Evaluation Protocol=Finetuning2022.08 | 82.4 | 95.4 | 97.9 | — | — | |
| BLIPModel Category=Dual encoder + Fusion encoder reranking, #Trainable Params=446M, Evaluation Protocol=Fine-tuned2023.01 | 82.4 | 95.4 | 97.9 | — | — | |
| OmniVLEvaluation Setting=Fine-tuning2023.05 | 82.1 | 95.9 | 98.1 | — | — | |
| BLIPPretrain data=129M2022.05 | 82.1 | 95.4 | 97.9 | — | — | |
| BLIPPre-train # Images=129M2022.01 | 81.9 | 95.4 | 97.8 | — | — | |
| FlorenceEvaluation Protocol=Fine-tuned2021.11 | 81.8 | 95.2 | — | — | — | |
| FlorenceArchitecture Type=Dual-encoder, Evaluation Protocol=Finetuning2022.08 | 81.8 | 95.2 | — | — | — | |
| FlorenceFine-tuned=true, # Pre-train images=900M2022.09 | 81.8 | 95.2 | — | — | — | |
| FlorenceEvaluation Setting=Fine-tuning2023.05 | 81.8 | 95.2 | — | — | — | |
| FlorencePre-training Images=900M, Parameters=893M, Evaluation Protocol=Fine-tuned2022.12 | 81.8 | 95.2 | — | — | — | |
| FlorenceModel Category=Dual-encoder models, #Trainable Params=893M, Evaluation Protocol=Fine-tuned2023.01 | 81.8 | 95.2 | — | — | — | |
| FlorencePretrain data=0.9B2022.05 | 81.8 | 95.2 | — | — | — | |
| PTP-BLIPPre-training Images=14M, Parameters=220M, Evaluation Protocol=Fine-tuned2022.12 | 81.5 | 95.9 | 97.9 | — | — | |
| X-VLM oursPre-train=true, Data size=4M2022.10 | 81.4 | 95.6 | 97.9 | — | — | |
| BLIP_CapFilt-LPre-train # Images=129M2022.01 | 81.2 | 95.7 | 97.9 | — | — | |
| BLIPPre-train # Images=14M2022.01 | 80.6 | 95.2 | 97.6 | — | — | |
| BLIP#PT Data=14M2022.12 | 80.6 | 95.2 | 97.6 | — | — | |
| BLIPPre-training Images=14M, Parameters=220M, Evaluation Protocol=Fine-tuned2022.12 | 80.6 | 95.2 | 97.6 | — | — | |
| BLIPPretrain data=14M2022.05 | 80.6 | 95.2 | 97.6 | — | — | |
| BLIP2026.03 | 80.6 | 95.2 | — | — | — | |
| X-VLMPre-train=true, Data size=4M2022.10 | 80.4 | 95.5 | 98.2 | — | — | |
| FIBER + EQSIMFine-tuning=COCO, Resolution=288x2882023.03 | 80.16 | 95.44 | 97.69 | — | — | |
| FLIPEvaluation Protocol=Fine-tuned2021.11 | 78.9 | 94.4 | — | — | — | |
| FILIPArchitecture Type=Dual-encoder, Evaluation Protocol=Finetuning2022.08 | 78.9 | 94.4 | 97.4 | — | — | |
| FILIPFine-tuned=true, # Pre-train images=340M2022.09 | 78.9 | 94.4 | 97.4 | — | — | |
| FILIPEvaluation Setting=Fine-tuning2023.05 | 78.9 | 94.4 | 97.4 | — | — | |
| FILIPPre-training Images=340M, Parameters=787M, Evaluation Protocol=Fine-tuned2022.12 | 78.9 | 94.4 | 97.4 | — | — | |
| FILIPModel Category=Dual-encoder models, #Trainable Params=417M, Evaluation Protocol=Fine-tuned2023.01 | 78.9 | 94.4 | 97.4 | — | — | |
| BiCro*Noise=20%, mismatch_threshold=true2023.03 | 78.8 | 96.1 | 98.6 | — | — | |
| BiCroNoise=20%2023.03 | 78.2 | 95.9 | 98.4 | — | — | |
| VLMoPretrain data=4M2022.05 | 78.2 | 94.4 | 97.4 | — | — | |
| SEPSVisual Encoder=ViT-Base-384, Textual Encoder=BERT-base, Image Resolution=384x384, Patch Size=24x24, Fine-Grained Alignment (FG)=true2025.11 | 77.8 | 88.7 | 94.8 | — | 534.6 | |
| ALBEFPre-train # Images=14M2022.01 | 77.6 | 94.3 | 97.2 | — | — | |
| ALBEFArchitecture Type=Dual encoder + Fusion encoder reranking, Evaluation Protocol=Finetuning2022.08 | 77.6 | 94.3 | 97.2 | — | — | |
| ALBEFFine-tuned=true, # Pre-train images=14M2022.09 | 77.6 | 94.3 | 97.2 | — | — | |
| ALBEFPre-train=true, Data size=14M2022.10 | 77.6 | 94.3 | 97.2 | — | — | |
| PTP-BLIPPre-training Images=4M, Parameters=220M, Evaluation Protocol=Fine-tuned2022.12 | 77.6 | 94.2 | 97 | — | — | |
| ALBEFPre-training Images=14M, Parameters=210M, Evaluation Protocol=Fine-tuned2022.12 | 77.6 | 94.3 | 97.2 | — | — | |
| ALBEFModel Category=Dual encoder + Fusion encoder reranking, #Trainable Params=233M, Evaluation Protocol=Fine-tuned2023.01 | 77.6 | 94.3 | 97.2 | — | — | |
| ALBEFPretrain data=14M2022.05 | 77.6 | 94.3 | 97.2 | — | — | |
| ALBEF2026.03 | 77.6 | 94.3 | — | — | — | |
| DECLNoise=20%2023.03 | 77.5 | 95.9 | 98.4 | — | — | |
| ERNIE-ViL 2.0Fine-tuned=true, # Pre-train images=29M2022.09 | 77.4 | 93.6 | 97.1 | — | — | |
| ALIGNEvaluation Protocol=Fine-tuned2021.02 | 77 | 93.5 | 96.9 | — | — | |
| ALIGNEvaluation Protocol=Fine-tuned2021.11 | 77 | 93.5 | — | — | — | |
| ALIGNPre-train # Images=1.8B2022.01 | 77 | 93.5 | 96.9 | — | — | |
| ALIGNArchitecture Type=Dual-encoder, Evaluation Protocol=Finetuning2022.08 | 77 | 93.5 | 96.9 | — | — | |
| ALIGNFine-tuned=true, # Pre-train images=1.8B2022.09 | 77 | 93.5 | 96.9 | — | — | |
| ALIGN#PT Data=1.2B2022.12 | 77 | 93.5 | 96.9 | — | — | |
| ALIGNEvaluation Setting=Fine-tuning2023.05 | 77 | 93.5 | 96.9 | — | — | |
| ALIGNPre-training Images=1.8B, Parameters=820M, Evaluation Protocol=Fine-tuned2022.12 | 77 | 93.5 | 96.9 | — | — | |
| BiCro*Noise=40%, mismatch_threshold=true2023.03 | 77 | 95.9 | 98.3 | — | — | |
| ALIGNModel Category=Dual-encoder models, #Trainable Params=820M, Evaluation Protocol=Fine-tuned2023.01 | 77 | 93.5 | 96.9 | — | — | |
| ALIGNPretrain data=1.8B2022.05 | 77 | 93.5 | 96.9 | — | — | |
| NCRNoise=20%2023.03 | 76.6 | 95.6 | 98.2 | — | — | |
| BiCroNoise=40%2023.03 | 76.4 | 95.2 | 98.6 | — | — | |
| FIBERFine-tuning=COCO, Resolution=288x2882023.03 | 75.88 | 93.92 | 96.79 | — | — | |
| DECLNoise=40%2023.03 | 75.6 | 95.5 | 98.3 | — | — | |
| VinVLArchitecture Type=Fusion-encoder, Evaluation Protocol=Finetuning2022.08 | 75.4 | 92.9 | 96.2 | — | — | |
| VinVLPre-train=true, Data size=5.6M2022.10 | 75.4 | 92.9 | 96.2 | — | — | |
| VinVLModel Category=Fusion-encoder models, #Trainable Params=345M, Evaluation Protocol=Fine-tuned2023.01 | 75.4 | 92.9 | 96.2 | — | — | |
| BLIPPre-training Images=4M, Parameters=220M, Evaluation Protocol=Fine-tuned, Implementation=Own implementation2022.12 | 75.2 | 93.3 | 96.3 | — | — | |
| NCRNoise=40%2023.03 | 74.7 | 94.6 | 98 | — | — | |
| SEPSVisual Encoder=Swin-Base-384, Textual Encoder=BERT-base, Image Resolution=384x384, Patch Size=12x12, Fine-Grained Alignment (FG)=true2025.11 | 74.7 | 88.4 | 94.3 | — | 519.1 | |
| VinVL-Base# PT Pairs=8.9M, Architecture=Single-Stream, Visual Encoder Pre-training=standard, Ensemble=false2022.04 | 74.6 | 92.6 | 96.3 | — | — | |
| VinVLPre-training Images=4M, Parameters=157M, Evaluation Protocol=Fine-tuned2022.12 | 74.6 | 92.6 | 96.3 | — | — | |
| BiCroNoise=60%2023.03 | 73.9 | 94.7 | 97.7 | — | — | |
| BiCro*Noise=60%, mismatch_threshold=true2023.03 | 73.9 | 94.4 | 97.8 | — | — | |
| SEPSVisual Encoder=ViT-Base-224, Textual Encoder=BERT-base, Image Resolution=224x224, Patch Size=14x14, Fine-Grained Alignment (FG)=true2025.11 | 73.9 | 85.2 | 92.1 | — | 516.9 | |
| OscarEvaluation Protocol=Fine-tuned2021.02 | 73.5 | 92.2 | 96 | — | — | |
| OscarEvaluation Protocol=Fine-tuned2021.11 | 73.5 | 92.2 | — | — | — | |
| OscarArchitecture Type=Fusion-encoder, Evaluation Protocol=Finetuning2022.08 | 73.5 | 92.2 | 96 | — | — | |
| OSCARFine-tuned=true, # Pre-train images=4M2022.09 | 73.5 | 92.2 | 96 | — | — | |
| FIBER2023.03 | 73.39 | 92.59 | 96.41 | — | — | |
| CDDSBackbone=ViT-224-Large + BERT-Large, Patch size=16×162026.03 | 73.3 | 92.2 | — | — | — | |
| ALBEF#PT Data=4M2022.12 | 73.1 | 91.4 | 96 | — | — | |
| DECLNoise=60%2023.03 | 73 | 94.2 | 97.9 | — | — | |
| LAPSBackbone=ViT-224-Large + BERT-Large, Patch size=16×162026.03 | 72.9 | 91.7 | — | — | — | |
| HiTeA#PT Data=5M2022.12 | 72.4 | 90.9 | 95.4 | — | — | |
| VSE∞(WSL) oursPre-train=false2022.10 | 71.9 | 92 | 95.9 | — | — | |
| Singularity#PT Data=5M2022.12 | 71.9 | 90.8 | 95.4 | — | — | |
| SEPSVisual Encoder=Swin-Base-224, Textual Encoder=BERT-base, Image Resolution=224x224, Patch Size=7x7, Fine-Grained Alignment (FG)=true2025.11 | 71.9 | 86 | 92.4 | — | 506.1 | |
| COOKIE# PT Pairs=5.9M, Architecture=Two-Stream, Visual Encoder Pre-training=940M tagged images, Ensemble=true2022.04 | 71.6 | 90.9 | 95.4 | — | — | |
| SAFNoise=20%2023.03 | 71.5 | 94 | 97.5 | — | — | |
| PTP-BLIP#Images=14M, Parameters=220M, zero-shot=true2022.12 | 71.4 | 91.3 | 95.5 | — | — | |
| MaMMUTimage model size=630M, evaluation protocol=zero-shot, model type=dual-encoder2023.03 | 70.7 | 89.1 | 93.7 | — | — | |
| COTS# PT Pairs=15.3M, Architecture=Two-Stream, Visual Encoder Pre-training=standard, Ensemble=true2022.04 | 70.6 | 91 | 95.3 | — | — | |
| CDDSBackbone=ViT-224 + BERT, Patch size=14×142026.03 | 70.4 | 91 | — | — | — | |
| OSCARPre-train # Images=4M2022.01 | 70 | 91.1 | 95.5 | — | — | |
| OSCAR-Base# PT Pairs=6.5M, Architecture=Single-Stream, Visual Encoder Pre-training=standard, Ensemble=false2022.04 | 70 | 91.1 | 95.5 | — | — | |
| OSCAR#PT Data=4M2022.12 | 70 | 91.1 | 95.5 | — | — |