Image Captioning on MS COCO (Karpathy)
140.9CIDEr-DVinVL_large
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| VinVL_largePre-training Images Scale=5.6M, CIDEr-optimized fine-tuning=true2021.11 | 140.9 | 41 | — | — | — | |
| X-VLMPre-training Images Scale=4M, CIDEr-optimized fine-tuning=true2021.11 | 140.8 | 41.3 | — | — | — | |
| VinVLbVisual Encoder=RESNEXt152, Pre-training Data (# I-T)=9M, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 140.4 | 40.9 | 30.9 | — | 25.1 | |
| X-VLMPre-training Images Scale=16M, CIDEr-optimized fine-tuning=true2021.11 | 140.3 | 41 | — | — | — | |
| ViTCAP*Visual Encoder=ViTb, Pre-training Data (# I-T)=10M, Optimization Objective=CIDEr Optimization, Architecture=Detector-free, Pre-training Status=With VLP2021.12 | 138.1 | 41.2 | 30.1 | 60.1 | 24.1 | |
| OSCARbVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=7M, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 137.6 | 40.5 | 29.7 | — | 22.8 | |
| RSTNetVisual Encoder=RESNEXT152, Pre-training Data (# I-T)=None, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 135.6 | 40.1 | 29.8 | 59.5 | 23.3 | |
| SimVLM_basePre-training Images Scale=1.8B2021.11 | 134.8 | 39 | — | — | — | |
| X-VLMPre-training Images Scale=16M, CIDEr-optimized fine-tuning=false2021.11 | 134 | 39.9 | — | — | — | |
| ViTCAPVisual Encoder=ViTb, Pre-training Data (# I-T)=None, Optimization Objective=CIDEr Optimization, Architecture=Detector-free, Pre-training Status=Without VLP2021.12 | 133.1 | 40.1 | 29.4 | 59.4 | 23 | |
| X-VLMPre-training Images Scale=4M, CIDEr-optimized fine-tuning=false2021.11 | 133.1 | 39.8 | — | — | — | |
| X-LANVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 132 | 39.5 | 29.5 | 59.2 | 23.4 | |
| Transformer+CATTBackbone=Transformer2021.03 | 131.7 | 39.4 | 29.3 | 58.9 | 22.8 | |
| MiniVLMVisual Encoder=Eff-DET, Pre-training Data (# I-T)=14M, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 131.7 | 39.2 | 29.7 | — | 23.5 | |
| M2TransformerBackbone=Transformer2021.03 | 131.2 | 39.1 | 29.2 | 58.6 | 22.6 | |
| M² Transfm.Visual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 131.2 | 39.1 | 29.2 | 58.6 | 22.6 | |
| AoANetBackbone=Transformer2021.03 | 129.8 | 38.9 | 29.2 | 58.8 | 22.4 | |
| AoANetVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 129.8 | 38.9 | 29.2 | 58.8 | 22.4 | |
| VLPBackbone=Transformer2021.03 | 129.3 | 39.5 | — | — | 23.2 | |
| UVLPVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=4M, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 129.3 | 39.5 | 29.3 | — | 23.2 | |
| VinVLbVisual Encoder=RESNEXt152, Pre-training Data (# I-T)=9M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 129.3 | 38.2 | 30.3 | — | 23.6 | |
| TransformerBackbone=Transformer2021.03 | 128.5 | 38.6 | 28.5 | 58.4 | 22 | |
| BUTD+CATTBackbone=LSTM2021.03 | 128.3 | 38.6 | 28.5 | 58.6 | 21.9 | |
| SGAEBackbone=LSTM2021.03 | 127.8 | 38.4 | 28.4 | 58.6 | 22.1 | |
| SGAEVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 127.8 | 38.4 | 28.4 | 58.6 | 22.1 | |
| LBPFBackbone=LSTM2021.03 | 127.6 | 38.3 | 28.5 | 58.4 | 22 | |
| GCN-LSTMBackbone=LSTM2021.03 | 127.6 | 38.2 | 28.5 | 58.3 | 22 | |
| LBPFVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 127.6 | 38.3 | 28.5 | 58.4 | 22 | |
| BUTDBackbone=LSTM2021.03 | 125.3 | 37.2 | 27.5 | 57.3 | 21.1 | |
| ViTCAP*Visual Encoder=ViTb, Pre-training Data (# I-T)=10M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-free, Pre-training Status=With VLP2021.12 | 125.2 | 36.3 | 29.3 | 58.1 | 22.6 | |
| UNIMObVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=9M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 124.4 | 38.8 | — | — | — | |
| OSCARbVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=7M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 123.7 | 36.5 | 30.3 | — | 23.1 | |
| X-LANVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 122 | 38.2 | 28.8 | 58 | 21.9 | |
| RFNetVisual Encoder=Ensemble, Pre-training Data (# I-T)=None, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 121.9 | 36.5 | 27.7 | 57.3 | 21.2 | |
| ViTCAPVisual Encoder=ViTb, Pre-training Data (# I-T)=None, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-free, Pre-training Status=Without VLP2021.12 | 121.8 | 35.7 | 28.8 | 57.6 | 22.1 | |
| DistillVLMVisual Encoder=Eff-DET, Pre-training Data (# I-T)=7M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 120.8 | 35.6 | 28.7 | — | 22.1 | |
| BUTDVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=CIDEr Optimization, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 120.1 | 36.3 | 27.7 | 56.9 | 21.4 | |
| AoANetVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 119.8 | 37.2 | 28.4 | 57.5 | 21.3 | |
| MiniVLMVisual Encoder=Eff-DET, Pre-training Data (# I-T)=14M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 119.8 | 35.6 | 28.6 | — | 21.6 | |
| Unified VLPModel size=122M, Evaluation Protocol=fully supervised2021.10 | 117.7 | — | — | — | 21.3 | |
| E2E-VLPVisual Encoder=ResNet50, Pre-training Data (# I-T)=6M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-free, Pre-training Status=With VLP2021.12 | 117.3 | 36.2 | — | — | — | |
| E2E-VLP2021.11 | 117.3 | 36.2 | — | — | — | |
| UVLPVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=4M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 116.9 | 36.5 | 28.4 | — | 21.2 | |
| SGAEVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 116.7 | 36.9 | 27.7 | 57.2 | 20.9 | |
| VL-T5Visual Encoder=F-RCNN101, Pre-training Data (# I-T)=9M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=With VLP2021.12 | 116.5 | — | — | — | — | |
| LBPFVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 116.4 | 37.4 | 28.1 | 57.5 | 21.2 | |
| BUTDVisual Encoder=F-RCNN101, Pre-training Data (# I-T)=None, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 113.5 | 36.2 | 27 | 56.4 | 20.3 | |
| ViLT-CAPVisual Encoder=ViTb, Pre-training Data (# I-T)=10M, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-free, Pre-training Status=With VLP, Note=Results achieved using ViLT [33] pre-trained checkpoint2021.12 | 113.5 | 33.7 | 27.7 | 56.1 | 20.9 | |
| RFNetVisual Encoder=Ensemble, Pre-training Data (# I-T)=None, Optimization Objective=Cross-Entropy Loss, Architecture=Detector-based, Pre-training Status=Without VLP2021.12 | 112.5 | 35.8 | 27.4 | 56.5 | 20.5 | |
| SimVLM_hugeEvaluation Protocol=Zero-shot2021.10 | 102.3 | — | — | — | 22.1 | |
| FEWVLM_largeModel size=740M, Evaluation Protocol=Few-shot2021.10 | 100.4 | — | — | — | 19.1 | |
| FEWVLM_baseModel size=224M, Evaluation Protocol=Few-shot2021.10 | 98.7 | — | — | — | 18.9 | |
| FEWVLM_largeModel size=740M, Evaluation Protocol=Zero-shot2021.10 | 92.1 | — | — | — | 17.3 | |
| FEWVLM_baseModel size=224M, Evaluation Protocol=Zero-shot2021.10 | 84.5 | — | — | — | 16.2 | |
| VL-T5_no-vqaModel size=224M, Evaluation Protocol=Few-shot2021.10 | 43 | — | — | — | 10.8 | |
| VL-T5_no-vqaModel size=224M, Evaluation Protocol=Zero-shot2021.10 | 4.9 | — | — | — | 2 |