Image Captioning on MS COCO (test)
140.4CIDErVinVLbase
Evaluation results
| Method | Links | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VinVLbase2021.07 | 140.4 | — | — | — | 40.9 | 30.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 25.1 | |
| OSCARLOptimization Strategy=CIDEr optimization2020.04 | 140 | — | — | — | 41.7 | 30.6 | 24.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OSCARLOptimization Strategy=CIDEr optimization2020.04 | 140 | — | — | — | 41.7 | 30.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 24.5 | |
| OSCARBOptimization Strategy=CIDEr optimization2020.04 | 137.6 | — | — | — | 40.5 | 29.7 | 22.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OSCARBOptimization Strategy=CIDEr optimization2020.04 | 137.6 | — | — | — | 40.5 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.8 | |
| Oscarbase2021.07 | 137.6 | — | — | — | 40.5 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.8 | |
| CLIP-Res50x4 TransformerVisual Encoder=CLIP-ResNet-50x42021.07 | 134.2 | — | — | — | 40.2 | 29.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 23.8 | |
| CLIP-Res101 TransformerVisual Encoder=CLIP-ResNet-1012021.07 | 130.3 | — | — | — | 39.2 | 29.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 23 | |
| AoANetOptimization Strategy=CIDEr optimization2020.04 | 129.8 | — | — | — | 38.9 | 29.2 | 22.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AoANetOptimization Strategy=CIDEr optimization2020.04 | 129.8 | — | — | — | 38.9 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.4 | |
| VLP2021.07 | 129.8 | — | — | — | 39.5 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.4 | |
| AoANet2021.07 | 129.8 | — | — | — | 38.9 | 29.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.4 | |
| VLPOptimization Strategy=CIDEr optimization2020.04 | 129.3 | — | — | — | 39.5 | 29.3 | 23.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VLPOptimization Strategy=CIDEr optimization2020.04 | 129.3 | — | — | — | 39.5 | 29.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 23.2 | |
| CLIP-Res50 TransformerVisual Encoder=CLIP-ResNet-502021.07 | 127.9 | — | — | — | 38.6 | 28.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.7 | |
| OSCARLOptimization Strategy=cross-entropy optimization2020.04 | 127.8 | — | — | — | 37.4 | 30.7 | 23.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OSCARLOptimization Strategy=cross-entropy optimization2020.04 | 127.8 | — | — | — | 37.4 | 30.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 23.5 | |
| BUTDTransformer*2021.07 | 127.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.5 | |
| BIAModel=LLaVA 1.5-7B, Prompt=Provide a short caption for this image2025.06 | 124.9 | — | — | — | 35 | 28.6 | 57.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.7 | |
| OSCARBOptimization Strategy=cross-entropy optimization2020.04 | 123.7 | — | — | — | 36.5 | 30.3 | 23.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OSCARBOptimization Strategy=cross-entropy optimization2020.04 | 123.7 | — | — | — | 36.5 | 30.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 23.1 | |
| BIA + SCGAModel=LLaVA 1.5-7B, Prompt=Provide a short caption for this image2025.06 | 123.1 | — | — | — | 34.5 | 28.4 | 57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.2 | |
| BenignModel=LLaVA 1.5-7B, Prompt=Provide a short caption for this image2025.06 | 122.7 | — | — | — | 34.9 | 29.1 | 58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22.9 | |
| ImageNet-Res101 TransformerVisual Encoder=ResNet-101 (ImageNet)2021.07 | 121.1 | — | — | — | 36.8 | 27.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 21.5 | |
| BUTDOptimization Strategy=CIDEr optimization2020.04 | 120.1 | — | — | — | 36.3 | 27.7 | 21.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BUTDOptimization Strategy=CIDEr optimization2020.04 | 120.1 | — | — | — | 36.3 | 27.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 21.4 | |
| BUTD2021.07 | 120.1 | — | — | — | 36.3 | 27.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 21.4 | |
| AoANetOptimization Strategy=cross-entropy optimization2020.04 | 119.8 | — | — | — | 37.2 | 28.4 | 21.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AoANetOptimization Strategy=cross-entropy optimization2020.04 | 119.8 | — | — | — | 37.2 | 28.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 21.3 | |
| ImageNet-Res50 TransformerVisual Encoder=ResNet-50 (ImageNet)2021.07 | 118.8 | — | — | — | 36.2 | 27.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 21.2 | |
| VLPOptimization Strategy=cross-entropy optimization2020.04 | 117.7 | — | — | — | 36.5 | 28.4 | 21.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VLPOptimization Strategy=cross-entropy optimization2020.04 | 117.7 | — | — | — | 36.5 | 28.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 21.3 | |
| IDEFICS-80BLLM=Llama65B, Trainable Parameters=14B, Shots=322023.11 | 116.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flamingo-80BLLM=Chinchilla70B, Trainable Parameters=10B, Shots=322023.11 | 113.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BUTDOptimization Strategy=cross-entropy optimization2020.04 | 113.5 | — | — | — | 36.2 | 27 | 20.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BUTDOptimization Strategy=cross-entropy optimization2020.04 | 113.5 | — | — | — | 36.2 | 27 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 20.3 | |
| IDEFICS-80BLLM=Llama65B, Trainable Parameters=14B, Shots=42023.11 | 110.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SceneTrilogystatus=Proposed2022.04 | 107 | 76 | — | — | 35.9 | 26.9 | 56.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 20.9 | |
| CrossCap2022.04 | 106.7 | 75.5 | — | — | 34.3 | 26.1 | 55.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 20.1 | |
| Flamingo-9BLLM=Chinchilla7B, Trainable Parameters=2B, Shots=322023.11 | 106.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MulCap2022.04 | 106 | 74.9 | — | — | 33.2 | 25.5 | 54.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 19.5 | |
| LNFMM2022.04 | 105.5 | 74.7 | — | — | 31.8 | 24.7 | 53.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 18.8 | |
| De-DiffusionVisual Encoder=ViT-L, LLM=PaLM 2-L, Trainable Parameters=135M, Shots=322023.11 | 103.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flamingo-80BLLM=Chinchilla70B, Trainable Parameters=10B, Shots=42023.11 | 103.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| De-DiffusionVisual Encoder=ViT-L, LLM=PaLM 2-L, Trainable Parameters=135M, Shots=42023.11 | 100.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AG-CVAE2022.04 | 100.1 | 73.2 | — | — | 31.1 | 24.5 | 52.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 18.8 | |
| OpenFlamingo-9BLLM=MPT7B, Shots=322023.11 | 99.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GMM-CVAE2022.04 | 98.6 | 72.9 | — | — | 30.7 | 24.2 | 52.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 17.7 | |
| ATTreferences=40-Refs2016.03 | 96 | 90 | 82 | 71 | 60 | 34 | 68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AnyMALVisual Encoder=VIT-G, LLM=Llama270B, Trainable Parameters=0, Shots=0, In-domain COCO training=true2023.11 | 95.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Googlereferences=40-Refs2016.03 | 95 | 89 | 80 | 69 | 59 | 35 | 68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Googlereferences=5-Refs2016.03 | 94 | 71 | 54 | 41 | 31 | 25 | 53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ATTreferences=5-Refs2016.03 | 94 | 73 | 57 | 42 | 32 | 25 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSR Captivatorreferences=40-Refs2016.03 | 94 | 90 | 82 | 71 | 60 | 34 | 68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flamingo-9BLLM=Chinchilla7B, Trainable Parameters=2B, Shots=42023.11 | 93.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSR Captivatorreferences=5-Refs2016.03 | 93 | 72 | 54 | 41 | 31 | 25 | 53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Att-RegionCNN+LSTMreferences=40-Refs2016.03 | 93 | 89 | 80 | 69 | 58 | 33 | 67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSRreferences=40-Refs2016.03 | 93 | 88 | 79 | 68 | 57 | 33 | 66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Att-RegionCNN+LSTMreferences=5-Refs2016.03 | 92 | 73 | 56 | 41 | 31 | 25 | 53 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| De-DiffusionVisual Encoder=ViT-L, LLM=PaLM 2-S, Trainable Parameters=135M, Shots=322023.11 | 92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IDEFICS-80BLLM=Llama65B, Trainable Parameters=14B, Shots=02023.11 | 91.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSRreferences=5-Refs2016.03 | 91 | 70 | 53 | 39 | 29 | 25 | 52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Humanreferences=40-Refs2016.03 | 91 | 88 | 74 | 63 | 47 | 34 | 63 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BPE-MoS# Softmaxes=92018.09 | 89.26 | — | — | — | 30.06 | 24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Nearest Neighborreferences=5-Refs2016.03 | 89 | 70 | 52 | 38 | 28 | 24 | 51 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRCNreferences=40-Refs2016.03 | 89 | 87 | 77 | 65 | 53 | 32 | 66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Montrealreferences=40-Refs2016.03 | 89 | 88 | 78 | 66 | 54 | 32 | 65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenFlamingo-9BLLM=MPT7B, Shots=42023.11 | 89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid-LightRNN-MoS# Softmaxes=92018.09 | 88.96 | — | — | — | 30.02 | 23.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Baseline# Softmaxes=12018.09 | 88.5 | — | — | — | 29.64 | 23.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| De-DiffusionVisual Encoder=ViT-L, LLM=PaLM 2-L, Trainable Parameters=135M, Shots=02023.11 | 88.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| De-DiffusionVisual Encoder=ViT-L, LLM=PaLM 2-S, Trainable Parameters=135M, Shots=42023.11 | 87.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LRCNreferences=5-Refs2016.03 | 87 | 70 | 53 | 38 | 28 | 24 | 52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Montrealreferences=5-Refs2016.03 | 87 | 71 | 53 | 38 | 28 | 24 | 52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Humanreferences=5-Refs2016.03 | 85 | 66 | 47 | 32 | 22 | 20 | 48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flamingo-80BLLM=Chinchilla70B, Trainable Parameters=10B, Shots=02023.11 | 84.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Nearest Neighborreferences=40-Refs2016.03 | 82 | 85 | 75 | 63 | 52 | 29 | 61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VisualGPT (Our SRAU)PLM=GPT, Training Data=1%2021.02 | 80.9 | 69.5 | — | — | 25.6 | 22.6 | 49.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VisualGPT (Normalized SRAU)PLM=GPT, Training Data=1%2021.02 | 80.6 | 68.7 | — | — | 25.2 | 22.3 | 49.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerPLM=GPT, Training Data=1%2021.02 | 80.5 | 68.5 | — | — | 25.1 | 22.1 | 49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| M² TransformerPLM=GPT, Training Data=1%2021.02 | 80.4 | 68.2 | — | — | 25 | 22.4 | 49.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OpenFlamingo-9BLLM=MPT7B, Shots=02023.11 | 79.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flamingo-9Bshots=0-shot2023.09 | 79.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flamingo-9BLLM=Chinchilla7B, Trainable Parameters=2B, Shots=02023.11 | 79.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| m-RNNreferences=5-Refs2016.03 | 79 | 68 | 51 | 37 | 27 | 23 | 50 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| m-RNNreferences=40-Refs2016.03 | 79 | 87 | 76 | 64 | 53 | 30 | 64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OSCARPLM=BERT, Training Data=1%2021.02 | 78.4 | 67.2 | — | — | 23.3 | 22.5 | 49.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AoA TransformerPLM=GPT, Training Data=1%2021.02 | 78.4 | 68.5 | — | — | 24.6 | 22 | 48.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AoA TransformerPLM=None, Training Data=1%2021.02 | 75.5 | 67.6 | — | — | 23.6 | 21.5 | 48.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MLBLreferences=40-Refs2016.03 | 75 | 87 | 77 | 66 | 54 | 32 | 65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| X-TransformerPLM=None, Training Data=1%2021.02 | 74.1 | 67 | — | — | 23.6 | 21.2 | 48.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MLBLreferences=5-Refs2016.03 | 74 | 67 | 50 | 36 | 26 | 22 | 50 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| M² TransformerPLM=None, Training Data=1%2021.02 | 73 | 67.1 | — | — | 23.4 | 21.3 | 48.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerPLM=None, Training Data=1%2021.02 | 71.9 | 66 | — | — | 21.9 | 21.1 | 47.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VisualGPT (Our SRAU)PLM=GPT, Training Data=0.5%2021.02 | 70.3 | 66.2 | — | — | 22.1 | 21.1 | 47.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerPLM=GPT, Training Data=0.5%2021.02 | 69.5 | 65.1 | — | — | 21.8 | 20.6 | 46.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VisualGPT (Normalized SRAU)PLM=GPT, Training Data=0.5%2021.02 | 69.3 | 65.3 | — | — | 21.8 | 20.9 | 47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| NeuralTalkreferences=40-Refs2016.03 | 69 | 83 | 70 | 57 | 45 | 28 | 60 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| M² TransformerPLM=GPT, Training Data=0.5%2021.02 | 68.5 | 64.7 | — | — | 21.8 | 20.7 | 47.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AoA TransformerPLM=GPT, Training Data=0.5%2021.02 | 67.2 | 64.2 | — | — | 21.2 | 20.5 | 46.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |