Image Captioning on Flickr30K (Karpathy split)
123.1CIDErInstructBLIP
Evaluation Results
| Method | Links | |
|---|---|---|
| InstructBLIPvisual encoder=EVA-g, glue layer=QFormer, LLM=Vicuna-7B, Res.=224, param=188M2023.12 | 123.1 | |
| GITShot=full2022.05 | 98.5 | |
| GIT2Shot=full2022.05 | 98.5 | |
| GITShot=full2022.05 | 98.5 | |
| CogVLMTrain Data=1.5B, Protocol=zero-shot2023.11 | 94.9 | |
| GITLShot=full2022.05 | 92.4 | |
| InternVL-Chatvisual encoder=IViT-6B, glue layer=QLLAMA, LLM=Vicuna-13B, Res.=336, param=13B2023.12 | 92.2 | |
| InternVL-Chatvisual encoder=IViT-6B, glue layer=QLLAMA, LLM=Vicuna-13B, Res.=224, param=90M2023.12 | 89.9 | |
| InternVL-Chatvisual encoder=IViT-6B, glue layer=QLLAMA, LLM=Vicuna-7B, Res.=224, param=64M2023.12 | 89.7 | |
| GIT2Shot=290 (1%)2022.05 | 88.2 | |
| ASMglue layer=QFormer, LLM decoder=Husky-7B, Zero-shot=true2023.12 | 87.7 | |
| GITShot=290 (1%)2022.05 | 86.6 | |
| GITShot=290 (1%)2022.05 | 86.6 | |
| Qwen-VLzero-shot=true, backbone=Qwen-7B2023.08 | 85.8 | |
| Qwen-VLglue layer=VL-Adapter, LLM decoder=Qwen-7B, Zero-shot=true2023.12 | 85.8 | |
| Qwen-VLvisual encoder=CLIP-G, glue layer=VL-Adapter, LLM=Qwen-7B, Res.=448, param=9.6B2023.12 | 85.8 | |
| Qwen-VLTrain Data=1.4B, Protocol=zero-shot2023.11 | 85.8 | |
| InstructBLIP-FlanT5-XLmode=specialist2023.08 | 84.5 | |
| InstructBLIPzero-shot=true, backbone=Vicuna-13B2023.08 | 82.8 | |
| InstructBLIPglue layer=QFormer, LLM decoder=Vicuna-13B, Zero-shot=true2023.12 | 82.8 | |
| InstructBLIPvisual encoder=EVA-g, glue layer=QFormer, LLM=Vicuna-13B, Res.=224, param=188M2023.12 | 82.8 | |
| InstructBLIPTrain Data=129M, Protocol=zero-shot2023.11 | 82.4 | |
| GIT2Shot=322022.05 | 82 | |
| GITBShot=full2022.05 | 81.8 | |
| Qwen-VL-Chatzero-shot=true2023.08 | 81 | |
| Qwen-VL-Chatglue layer=VL-Adapter, LLM decoder=Qwen-7B, Zero-shot=true2023.12 | 81 | |
| Qwen-VL-Chatvisual encoder=CLIP-G, glue layer=VL-Adapter, LLM=Qwen-7B, Res.=448, param=9.6B2023.12 | 81 | |
| Kosmos-2zero-shot=true2023.08 | 80.5 | |
| GITShot=322022.05 | 80.5 | |
| GITShot=322022.05 | 80.5 | |
| GIT2Shot=162022.05 | 79.6 | |
| InternVL-Gglue layer=Cross-Attn, LLM decoder=QLLaMA, Zero-shot=true2023.12 | 79.2 | |
| FlamingoShot=162022.05 | 78.9 | |
| FlamingoShot=162022.05 | 78.9 | |
| GITShot=162022.05 | 78 | |
| GITShot=162022.05 | 78 | |
| FlamingoShot=322022.05 | 75.4 | |
| GITLShot=290 (1%)2022.05 | 75.4 | |
| FlamingoShot=322022.05 | 75.4 | |
| Shikrazero-shot=true, backbone=Vicuna-13B2023.08 | 73.9 | |
| Shikra-13Bglue layer=Linear, LLM decoder=Vicuna-13B, Zero-shot=true2023.12 | 73.9 | |
| Shikravisual encoder=CLIP-L, glue layer=Linear, LLM=Vicuna-13B, Res.=224, param=7B2023.12 | 73.9 | |
| GITBShot=290 (1%)2022.05 | 71.8 | |
| BLIP-2zero-shot=true, backbone=Vicuna-13B2023.08 | 71.6 | |
| BLIP-2glue layer=QFormer, LLM decoder=Vicuna-13B, Zero-shot=true2023.12 | 71.6 | |
| BLIP-2visual encoder=EVA-g, glue layer=QFormer, LLM=Vicuna-13B, Res.=224, param=188M2023.12 | 71.6 | |
| Zhou et al. (2020)Shot=full2022.05 | 68.5 | |
| GITLShot=322022.05 | 68.5 | |
| Zhou et al. (2020)Shot=full2022.05 | 68.5 | |
| Flamingo-80Bzero-shot=true2023.08 | 67.2 | |
| Flamingo-80Bglue layer=Cross-Attn, LLM decoder=Chinchilla-70B, Zero-shot=true2023.12 | 67.2 | |
| FlamingoTrain Data=2.3B, Protocol=zero-shot2023.11 | 67.2 | |
| FlamingoShot=02022.05 | 67.2 | |
| FlamingoShot=02022.05 | 67.2 | |
| Kosmos-1zero-shot=true2023.08 | 67.1 | |
| KOSMOS-2Inference mode=Zero-shot, Instruction tuning=false, Beam size=5, Resolution=224x2242023.06 | 66.7 | |
| KOSMOS-2glue layer=Linear, LLM decoder=KOSMOS-1, Zero-shot=true2023.12 | 66.7 | |
| GITBShot=322022.05 | 66.4 | |
| GITBShot=162022.05 | 65.8 | |
| IDEFICS-80B-Ivisual encoder=CLIP-H, glue layer=Cross-Attn, LLM=LLaMA-65B, Res.=224, param=15B2023.12 | 65.3 | |
| KOSMOS-1Inference mode=Zero-shot, Instruction tuning=false, Beam size=5, Resolution=224x2242023.06 | 65.2 | |
| GITLShot=162022.05 | 64.4 | |
| Flamingo-9BInference mode=Zero-shot, Instruction tuning=false, Beam size=5, Resolution=224x2242023.06 | 61.5 | |
| Flamingo-9Bzero-shot=true2023.08 | 61.5 | |
| Flamingo-9Bglue layer=Cross-Attn, LLM decoder=Chinchilla-7B, Zero-shot=true2023.12 | 61.5 | |
| Flamingo-3BInference mode=Zero-shot, Instruction tuning=false, Beam size=5, Resolution=224x2242023.06 | 60.6 | |
| IDEFICS-80Bvisual encoder=CLIP-H, glue layer=Cross-Attn, LLM=LLaMA-65B, Res.=224, param=15B2023.12 | 53.7 | |
| GIT2Train Data=12.9B, Protocol=zero-shot2023.11 | 50.7 | |
| GIT2Shot=02022.05 | 50.7 | |
| GITTrain Data=0.8B, Protocol=zero-shot2023.11 | 49.6 | |
| GITShot=02022.05 | 49.6 | |
| GITShot=02022.05 | 49.6 | |
| METALMInference mode=Zero-shot, Instruction tuning=false, Beam size=5, Resolution=224x2242023.06 | 43.4 | |
| GITLShot=02022.05 | 39.2 | |
| GITBShot=02022.05 | 35.2 | |
| FewVLMInference mode=Zero-shot, Instruction tuning=false, Beam size=5, Resolution=224x2242023.06 | 31 |