Image-to-Text Retrieval on MSCOCO (1K test)
85.4R@1ImageBERT
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ImageBERTtraining_mode=fine-tuned2020.01 | 85.4 | 98.7 | 99.8 | — | — | — | |
| SOHOBackbone=R1012021.04 | 85.1 | 97.4 | 99.4 | — | — | — | |
| Unicoder-VLEvaluation Protocol=pre-train + fine-tune2019.08 | 84.3 | 97.3 | 99.3 | — | — | — | |
| Unicoder-VLtraining_mode=fine-tuned2020.01 | 84.3 | 97.3 | 99.3 | — | — | — | |
| Unicoder-VL2021.04 | 84.3 | 97.3 | 99.3 | — | — | — | |
| MaxMatchBackbone Architecture=Faster R-CNN + BERT, Ensemble=false2026.03 | 83 | 96.9 | 98.9 | — | — | 533.8 | |
| USERBackbone Architecture=Faster R-CNN + BERT, Ensemble=false2026.03 | 82.8 | 96.8 | 98.8 | — | — | 530.5 | |
| CMSFBackbone Architecture=Faster R-CNN + BERT, Ensemble=false2026.03 | 81.9 | 96.7 | 98.9 | — | — | 533.8 | |
| CHANBackbone Architecture=Faster R-CNN + BERT, Ensemble=false2026.03 | 81.4 | 96.9 | 98.9 | — | — | 532.6 | |
| HREMBackbone Architecture=Faster R-CNN + BERT, Ensemble=false2026.03 | 81.1 | 96.6 | 98.9 | — | — | 530.7 | |
| TERANBackbone Architecture=Faster R-CNN + BERT, Ensemble=true2026.03 | 80.2 | 96.6 | 99 | — | — | 531.9 | |
| VSE∞Backbone Architecture=Faster R-CNN + BERT, Ensemble=false2026.03 | 79.7 | 96.4 | 98.9 | — | — | 527.5 | |
| CMSFBackbone Architecture=Faster R-CNN + BiGRU, Ensemble=false2026.03 | 78.9 | 96.2 | 98.7 | — | — | 524.7 | |
| VSE∞Backbone Architecture=Faster R-CNN + BiGRU, Ensemble=false2026.03 | 78.5 | 96 | 98.7 | — | — | 520.8 | |
| NAAFBackbone Architecture=Faster R-CNN + BiGRU, Ensemble=false2026.03 | 78.1 | 96.1 | 98.6 | — | — | 521.2 | |
| VSRN++Backbone Architecture=Faster R-CNN + BERT, Ensemble=true2026.03 | 77.9 | 96 | 98.5 | — | — | 523.6 | |
| VSRN + AOQ2021.01 | 77.5 | — | — | — | 44.7 | — | |
| SCG2019.08 | 76.6 | 96.3 | 99.2 | — | — | — | |
| SCGtraining_mode=fine-tuned2020.01 | 76.6 | 96.3 | 99.2 | — | — | — | |
| PFAN2019.08 | 76.5 | 96.3 | 99 | — | — | — | |
| PFANtraining_mode=fine-tuned2020.01 | 76.5 | 96.3 | 99 | — | — | — | |
| VSRN2021.01 | 76.2 | — | — | — | 41.2 | — | |
| VSRNBackbone Architecture=Faster R-CNN + BiGRU, Ensemble=true2026.03 | 76.2 | 94.8 | 98.2 | — | — | 516.8 | |
| Unicoder-VLEvaluation Protocol=w/o pre-training2019.08 | 75.1 | 94.3 | 97.8 | — | — | — | |
| MMCABackbone Architecture=Faster R-CNN + BERT, Ensemble=false2026.03 | 74.8 | 95.6 | 97.7 | — | — | 514.7 | |
| SCANBackbone=Faster R-CNN, ResNet, Retrieval direction=t-i + i-t ensemble, Pooling type=LSE + AVG2018.03 | 72.7 | 94.8 | 98.4 | — | — | — | |
| SCAN2019.08 | 72.7 | 94.8 | 98.4 | — | — | — | |
| SCANtraining_mode=fine-tuned2020.01 | 72.7 | 94.8 | 98.4 | — | — | — | |
| SCANBackbone=R1012021.04 | 72.7 | 94.8 | 98.4 | — | — | — | |
| SCANBackbone Architecture=Faster R-CNN + BiGRU, Ensemble=true2026.03 | 72.7 | 94.8 | 98.4 | — | — | 507.9 | |
| SCANBackbone=Faster R-CNN, ResNet, Retrieval direction=Text-Image (t-i), Pooling type=AVG, lambda1=92018.03 | 70.9 | 94.5 | 97.8 | — | — | — | |
| SCOBackbone=ResNet2018.03 | 69.9 | 92.9 | 97.5 | — | — | — | |
| SCO2019.06 | 69.9 | 92.9 | 97.5 | — | — | — | |
| SCANBackbone=Faster R-CNN, ResNet, Retrieval direction=Image-Text (i-t), Pooling type=AVG, lambda1=42018.03 | 69.2 | 93.2 | 97.5 | — | — | — | |
| PVSENumber of embeddings (K)=32019.06 | 69.2 | 91.6 | 96.6 | — | — | — | |
| PVSEK=22021.01 | 69.2 | — | — | — | 42.8 | — | |
| PCMEvariant=stochastic2021.01 | 68.8 | — | — | — | 45 | — | |
| GXNBackbone=ResNet2018.03 | 68.5 | — | 97.9 | — | — | — | |
| GXN2019.06 | 68.5 | — | 97.9 | — | — | — | |
| SCANBackbone=Faster R-CNN, ResNet, Retrieval direction=Image-Text (i-t), Pooling type=LSE, lambda1=4, lambda2=202018.03 | 68.4 | 93.9 | 98 | — | — | — | |
| PCMEvariant=mu only2021.01 | 68 | — | — | — | 45 | — | |
| SCANBackbone=Faster R-CNN, ResNet, Retrieval direction=Text-Image (t-i), Pooling type=LSE, lambda1=9, lambda2=62018.03 | 67.5 | 92.9 | 97.6 | — | — | — | |
| PVSENumber of embeddings (K)=12019.06 | 66.7 | 91 | 96.2 | — | — | — | |
| PVSEK=12021.01 | 66.7 | — | — | — | 40.3 | — | |
| DPCBackbone=ResNet2018.03 | 65.6 | 89.8 | 95.5 | — | — | — | |
| Ours (ResNet-50) Stage IIVisual=ft ResNet-50, Textual=ft ResNet-50 (w2v init.)2017.11 | 65.6 | 89.8 | 95.5 | 1 | — | — | |
| VSE++2019.08 | 64.7 | — | 95.9 | — | — | — | |
| VSE++Backbone=ResNet2018.03 | 64.6 | — | 95.7 | — | — | — | |
| VSE++2019.06 | 64.6 | 90 | 95.7 | — | — | — | |
| VSE++Backbone=R1522021.04 | 64.6 | 90 | 95.7 | — | — | — | |
| VSE++2021.01 | 64.6 | — | — | — | — | — | |
| AltCLIPBackbone=ViT-L, Zero-shot=true, Stage=Contrastive Learning2022.11 | 62.8 | 88.8 | 95.5 | 82 | — | — | |
| CN-CLIPBackbone=ViT-L, Zero-shot=true2022.11 | 61 | 84.7 | 93.6 | 81 | — | — | |
| AltCLIPTBackbone=ViT-L, Zero-shot=true, Stage=Teacher Learning2022.11 | 60.7 | 86.3 | 93.4 | 78.4 | — | — | |
| Ours (VGG-19) Stage IIVisual=ft VGG-19, Textual=ft ResNet-50 (w2v init.)2017.11 | 59.4 | 86.2 | 92.9 | 1 | — | — | |
| Multimodal Contrastive TrainingBackbone=ResNet-50, Pre-training=Vision-and-Language (COCO)2021.04 | 58.4 | — | 93.4 | 1 | — | — | |
| VirTexBackbone=ResNet-50, Pre-training=Vision-and-Language (COCO)2021.04 | 58.1 | — | 93 | 1 | — | — | |
| IN-supBackbone=ResNet-50, Pre-training=Supervised (ImageNet-1K)2021.04 | 57.9 | — | 92.7 | 1 | — | — | |
| RRF-Net2019.06 | 56.4 | 85.3 | 91.5 | — | — | — | |
| RRF-NetVisual=fixed ResNet-152, Textual=w2v + HGLMM2017.11 | 56.4 | 85.3 | 91.5 | — | — | — | |
| R2D2Backbone=ViT-L, Zero-shot=true, Reported original results=true2022.11 | 56.4 | 85 | 93.1 | 80.5 | — | — | |
| CMPM2019.06 | 56.1 | 86.3 | 92.9 | — | — | — | |
| 2WayNetBackbone=VGG2018.03 | 55.8 | 75.2 | — | — | — | — | |
| 2WayNet2019.06 | 55.8 | 75.2 | — | — | — | — | |
| 2WayNetVisual=fixed VGG-16, Textual=feature from [17]2017.11 | 55.8 | 75.2 | — | — | — | — | |
| Unicoder-VLEvaluation Protocol=zero-shot2019.08 | 54.4 | 82.8 | 90.6 | — | — | — | |
| WukongBackbone=ViT-L, Zero-shot=true2022.11 | 53.4 | 80.2 | 90.1 | 75.1 | — | — | |
| SM-LSTMBackbone=VGG2018.03 | 53.2 | 83.1 | 91.5 | — | — | — | |
| sm-LSTMVisual=fixed VGG-19, Textual=ft RNN2017.11 | 53.2 | 83.1 | 91.5 | 1 | — | — | |
| Ours (ResNet-50) Stage IVisual=fixed ResNet-50, Textual=ft ResNet-50 (w2v init.)2017.11 | 52.2 | 80.4 | 88.7 | 1 | — | — | |
| MoCo-v2Backbone=ResNet-50, Pre-training=Self-supervised (MoCo-v2)2021.04 | 51.6 | — | 90 | 1 | — | — | |
| VQA-A2019.06 | 50.5 | 80.1 | 89.7 | — | — | — | |
| VQA-AVisual=fixed VGG-19, Textual=ft RNN2017.11 | 50.5 | 80.1 | 89.7 | — | — | — | |
| DSPE2019.06 | 50.1 | 79.7 | 89.2 | — | — | — | |
| DSPE2019.08 | 50.1 | 79.7 | 89.2 | — | — | — | |
| SPEVisual=fixed VGG-19, Textual=w2v + HGLMM2017.11 | 50.1 | 79.7 | 89.2 | — | — | — | |
| Order-embeddingsBackbone=VGG2018.03 | 46.7 | — | 88.9 | — | — | — | |
| Order2019.06 | 46.7 | — | 88.9 | — | — | — | |
| TaiyiBackbone=ViT-L, Zero-shot=true2022.11 | 46.6 | 76.3 | 88.6 | 72.2 | — | — | |
| Ours (VGG-19) Stage IVisual=fixed VGG-19, Textual=ft ResNet-50 (w2v init.)2017.11 | 46 | 75.6 | 85.3 | 2 | — | — | |
| HM-LSTMBackbone=R-CNN, AlexNet2018.03 | 43.9 | — | 87.8 | — | — | — | |
| HM-LSTMVisual=fixed CNN from [32], Textual=ft RNN2017.11 | 43.9 | — | 87.8 | 2 | — | — | |
| m-CNN2019.06 | 42.8 | 73.1 | 84.1 | — | — | — | |
| m-CNN2019.08 | 42.8 | 73.1 | 84.1 | — | — | — | |
| m-CNNVisual=ft VGG-19, Textual=4 × ft CNN2017.11 | 42.8 | 73.1 | 84.1 | 2 | — | — | |
| RNN-FVVisual=fixed VGG-19, Textual=feature from [17]2017.11 | 41.5 | 72 | 82.9 | 2 | — | — | |
| m-RNNVisual=fixed VGG-16, Textual=ft RNN2017.11 | 41 | 73 | 83.5 | 2 | — | — | |
| GMM-FV2019.06 | 39.4 | 67.9 | 80.9 | — | — | — | |
| GMM-FVVisual=fixed VGG-16, Textual=w2v + GMM + HGLMM2017.11 | 39.4 | 67.9 | 80.9 | 2 | — | — | |
| Our modeltest_set_size=1K2014.12 | 38.4 | 69.9 | 80.5 | 1 | — | — | |
| DVSABackbone=R-CNN, AlexNet2018.03 | 38.4 | 69.9 | 80.5 | — | — | — | |
| DVSA2019.06 | 38.4 | 69.9 | 80.5 | — | — | — | |
| DVSA2019.08 | 38.4 | 69.9 | 80.5 | — | — | — | |
| DVSAVisual=ft RCNN, Textual=w2v + ft RNN2017.11 | 38.4 | 69.9 | 80.5 | 1 | — | — | |
| Our modeltest_set_size=5K2014.12 | 16.5 | 39.2 | 52 | 9 | — | — | |
| CLIPBackbone=ViT-L, Zero-shot=true2022.11 | 1.8 | 6.7 | 11.9 | 5.4 | — | — |