Image Captioning on MS COCO Karpathy (test)
149.1CIDErPaLI
Evaluation results
| Method | Links | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PaLINumber of parameters=17B2022.09 | 149.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BEIT-3Number of parameters=1.9B2022.09 | 147.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OFA BaseEncoder Layers=6, Decoder Layers=62022.11 | 146.7 | — | — | — | 0.428 | 0.317 | — | — | — | — | — | 0.258 | — | — | — | — | — | — | — | 1 | — | — | — | |
| PaLINumber of parameters=15B2022.09 | 146.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP-2 ViT-g OPT2.7B#Trainable Params=1.1B, Fine-tuned=true2023.01 | 145.8 | — | — | — | 0.437 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaLINumber of parameters=3B2022.09 | 145.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OFANumber of parameters=0.9B2022.09 | 145.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OFA#Trainable Params=930M, Fine-tuned=true2023.01 | 145.3 | — | — | — | 0.439 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP-2 ViT-g OPT6.7B#Trainable Params=1.1B, Fine-tuned=true2023.01 | 145.2 | — | — | — | 0.435 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GIT2Number of parameters=5.1B2022.09 | 145 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GITNumber of parameters=0.7B2022.09 | 144.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP-2 ViT-g FlanT5XL#Trainable Params=1.1B, Fine-tuned=true2023.01 | 144.5 | — | — | — | 0.424 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FIBER-GOLD-B#Pretrain Images=4M, CIDEr optimization=true2022.06 | 144.4 | — | — | — | 0.434 | 0.313 | — | — | — | — | — | 0.246 | — | — | — | — | — | — | — | — | — | — | — | |
| CoCaNumber of parameters=2.1B2022.09 | 143.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimVLMhugePre-train #Images=1.8B2022.01 | 143.3 | — | — | — | 40.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimVLM Hugetrainable_parameters=632M2022.09 | 143.3 | — | — | — | 0.406 | 0.337 | — | — | — | — | — | 0.254 | — | — | — | — | — | — | — | — | — | — | — | |
| SimVLM2022.09 | 143.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimVLM#Trainable Params=~1.4B, Fine-tuned=true2023.01 | 143.3 | — | — | — | 0.406 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FIBER-B#Pretrain Images=4M, CIDEr optimization=true2022.06 | 142.8 | — | — | — | 0.428 | 0.31 | — | — | — | — | — | 0.243 | — | — | — | — | — | — | — | — | — | — | — | |
| LEMON-B#Pretrain Images=200M, CIDEr optimization=true2022.06 | 142.7 | — | — | — | 0.416 | 0.31 | — | — | — | — | — | 0.251 | — | — | — | — | — | — | — | — | — | — | — | |
| SimVLM_large# Params=783M, Pre-training Data=More Data2022.11 | 142.6 | — | — | — | 40.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FREE-V-OArchitecture=Early Exit, Speed=1.75x2025.06 | 142.5 | — | — | — | 0.419 | 25.2 | — | — | — | — | — | 30.8 | — | — | — | — | — | — | — | — | — | — | — | |
| OFA_large# Params=472M, Pre-training Data=More Data2022.11 | 142.2 | — | — | — | 42.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP-2-V-FTraining=Semi-supervised, Speed=1.00x2025.06 | 141.5 | — | — | — | 0.404 | 25.2 | — | — | — | — | — | 29.1 | — | — | — | — | — | — | — | — | — | — | — | |
| X-VLM#Pretrain Images=4M, CIDEr optimization=true2022.06 | 140.8 | — | — | — | 0.413 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VinVL-B#Pretrain Images=5.7M, CIDEr optimization=true2022.06 | 140.4 | — | — | — | 0.409 | 0.309 | — | — | — | — | — | 0.251 | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP-2-V-OTraining=Semi-supervised, Speed=1.07x2025.06 | 139.8 | — | — | — | 0.417 | 25.5 | — | — | — | — | — | 30.5 | — | — | — | — | — | — | — | — | — | — | — | |
| LEMON Hugetrainable_parameters=675M2022.09 | 139.1 | — | — | — | 0.415 | 0.308 | — | — | — | — | — | 0.241 | — | — | — | — | — | — | — | — | — | — | — | |
| LEMONNumber of parameters=0.7B2022.09 | 139.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| X2-VLM_large# Params=593M, Pre-training Data=More Data2022.11 | 139.1 | — | — | — | 42.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EVLGenLLM=Vicuna-7b, Data=104M, steps=250k, Zero-shot=true2023.10 | 139.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimVLMTraining=Semi-supervised2025.06 | 138.3 | — | — | — | 0.386 | 24.8 | — | — | — | — | — | 29.8 | — | — | — | — | — | — | — | — | — | — | — | |
| OFA_base# Params=182M, Pre-training Data=More Data2022.11 | 138.2 | — | — | — | 41 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VITCAP#Pretrain Images=4M, CIDEr optimization=true2022.06 | 138.1 | — | — | — | 0.412 | 0.301 | — | — | — | — | — | 0.241 | — | — | — | — | — | — | — | — | — | — | — | |
| FlamingoNumber of parameters=80B2022.09 | 138.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flamingo#Trainable Params=10.6B, Fine-tuned=true2023.01 | 138.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP-2LLM=Vicuna-7b, Data=104M, steps=330k, Zero-shot=true2023.10 | 138 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EVLGenLLM=Vicuna-7b, Data=11M, steps=150k, Zero-shot=true2023.10 | 138 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EVLGenLLM=Vicuna-7b, Data=104M, steps=150k, Zero-shot=true2023.10 | 137.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EVLGenLLM=Vicuna-7b, Data=104M, steps=90k, Zero-shot=true2023.10 | 137.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MuEArchitecture=Early Exit, Speed=1.41x2025.06 | 137.5 | — | — | — | 0.379 | 23.6 | — | — | — | — | — | 28.5 | — | — | — | — | — | — | — | — | — | — | — | |
| MuEBackbone=OFA Base2022.11 | 137 | — | — | — | 0.416 | 0.306 | — | — | — | — | — | 0.244 | — | — | — | — | — | — | — | -40.2 | — | — | — | |
| BLIPViT-LPre-train #Images=129M2022.01 | 136.7 | — | — | — | 40.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP#Trainable Params=446M, Fine-tuned=true2023.01 | 136.7 | — | — | — | 0.404 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| X2-VLM_large# Params=593M, Pre-training Data=4M2022.11 | 136.7 | — | — | — | 42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| X2-VLM_base# Params=255M, Pre-training Data=More Data2022.11 | 136.1 | — | — | — | 41.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LEMONlargePre-train #Images=200M2022.01 | 135.7 | — | — | — | 40.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SimVLM-B#Pretrain Images=1.8B, CIDEr optimization=false2022.06 | 134.8 | — | — | — | 0.39 | 0.329 | — | — | — | — | — | 0.24 | — | — | — | — | — | — | — | — | — | — | — | |
| SimVLM_base# Params=273M, Pre-training Data=More Data2022.11 | 134.8 | — | — | — | 39 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FlamingoTraining=Semi-supervised2025.06 | 134.1 | — | — | — | 0.385 | 24.1 | — | — | — | — | — | 27.8 | — | — | — | — | — | — | — | — | — | — | — | |
| OmniVL# Params=288M, Pre-training Data=More Data2022.11 | 133.9 | — | — | — | 39.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FIBER-GOLD-B#Pretrain Images=4M, CIDEr optimization=false2022.06 | 133.6 | — | — | — | 0.403 | 0.307 | — | — | — | — | — | 0.236 | — | — | — | — | — | — | — | — | — | — | — | |
| X2-VLM_base# Params=255M, Pre-training Data=4M2022.11 | 133.6 | — | — | — | 41 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LEMONbasePre-train #Images=200M2022.01 | 133.3 | — | — | — | 40.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIPCapFilt-LPre-train #Images=129M2022.01 | 133.3 | — | — | — | 39.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIPCapFilt-L#Pretrain Images=129M, CIDEr optimization=false2022.06 | 133.3 | — | — | — | 0.397 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LEMON-B#Pretrain Images=200M, CIDEr optimization=false2022.06 | 133.3 | — | — | — | 0.403 | 0.302 | — | — | — | — | — | 0.233 | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP CapFilt-Ltrainable_parameters=224M2022.09 | 133.3 | — | — | — | 0.397 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| X-VLM#Pretrain Images=4M, CIDEr optimization=false2022.06 | 133.1 | — | — | — | 0.398 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LeeBLIPArchitecture=Early Exit, Speed=1.59x2025.06 | 132 | — | — | — | 0.374 | 22.8 | — | — | — | — | — | 27.6 | — | — | — | — | — | — | — | — | — | — | — | |
| BLIPPre-train #Images=129M2022.01 | 131.4 | — | — | — | 39.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP_base# Params=240M, Pre-training Data=More Data2022.11 | 131.4 | — | — | — | 39.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OFATraining=Semi-supervised2025.06 | 130.3 | — | — | — | 0.375 | 25.2 | — | — | — | — | — | 31.1 | — | — | — | — | — | — | — | — | — | — | — | |
| BLIPPre-train #Images=14M2022.01 | 129.7 | — | — | — | 38.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VinVLPre-train #Images=5.7M2022.01 | 129.3 | — | — | — | 38.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VinVL-B#Pretrain Images=5.7M, CIDEr optimization=false2022.06 | 129.3 | — | — | — | 0.382 | 0.303 | — | — | — | — | — | 0.236 | — | — | — | — | — | — | — | — | — | — | — | |
| VinVL#Trainable Params=345M, Fine-tuned=true2023.01 | 129.3 | — | — | — | 0.382 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeeCapBackbone=OFA2022.11 | 129 | — | — | — | 0.387 | 0.291 | — | — | — | — | — | 0.225 | — | — | — | — | — | — | — | -38 | — | — | — | |
| OFA TinyEncoder Layers=4, Decoder Layers=42022.11 | 128.7 | — | — | — | 0.381 | 0.292 | — | — | — | — | — | 0.231 | — | — | — | — | — | — | — | -33 | — | — | — | |
| FIBER-B#Pretrain Images=4M, CIDEr optimization=false2022.06 | 128.4 | — | — | — | 0.391 | 0.304 | — | — | — | — | — | 0.231 | — | — | — | — | — | — | — | — | — | — | — | |
| METER-CLIP-B#Pretrain Images=4M, CIDEr optimization=false2022.06 | 128.2 | — | — | — | 0.388 | 0.3 | — | — | — | — | — | 0.23 | — | — | — | — | — | — | — | — | — | — | — | |
| OSCAR Largetrainable_parameters=338M2022.09 | 127.8 | — | — | — | 0.374 | 0.307 | — | — | — | — | — | 0.235 | — | — | — | — | — | — | — | — | — | — | — | |
| OSCAR#Trainable Params=345M, Fine-tuned=true2023.01 | 127.8 | — | — | — | 0.374 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CaMELtrainable_parameters=76M2022.09 | 125.7 | — | — | — | 0.391 | 0.294 | — | — | — | — | — | 0.222 | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP-2 + MAFAEvaluation Protocol=Zero-shot2023.12 | 125.4 | — | — | — | 0.376 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VITCAP#Pretrain Images=4M, CIDEr optimization=false2022.06 | 125.2 | — | — | — | 0.363 | 0.293 | — | — | — | — | — | 0.226 | — | — | — | — | — | — | — | — | — | — | — | |
| UFO-B#Pretrain Images=4M, CIDEr optimization=false2022.06 | 122.8 | — | — | — | 0.36 | 0.289 | — | — | — | — | — | 0.222 | — | — | — | — | — | — | — | — | — | — | — | |
| SMALLCAP d=16, Largetrainable_parameters=47M, dimensionality=16, decoder=Large2022.09 | 121.8 | — | — | — | 0.372 | 0.283 | — | — | — | — | — | 0.215 | — | — | — | — | — | — | — | — | — | — | — | |
| SMALLCAP d=16, Medtrainable_parameters=22M, dimensionality=16, decoder=Medium2022.09 | 120.7 | — | — | — | 0.365 | 0.281 | — | — | — | — | — | 0.216 | — | — | — | — | — | — | — | — | — | — | — | |
| I-Tuning Mediumtrainable_parameters=44M2022.09 | 120 | — | — | — | 0.355 | 0.288 | — | — | — | — | — | 0.22 | — | — | — | — | — | — | — | — | — | — | — | |
| PABEE-BLIPArchitecture=Early Exit, Speed=1.45x2025.06 | 119.8 | — | — | — | 0.342 | 21.4 | — | — | — | — | — | 26.2 | — | — | — | — | — | — | — | — | — | — | — | |
| SMALLCAPtrainable_parameters=7M2022.09 | 119.7 | — | — | — | 0.37 | 0.279 | — | — | — | — | — | 0.213 | — | — | — | — | — | — | — | — | — | — | — | |
| I-Tuning Largetrainable_parameters=95M2022.09 | 119.4 | — | — | — | 0.348 | 0.293 | — | — | — | — | — | 0.224 | — | — | — | — | — | — | — | — | — | — | — | |
| SMALLCAP d=8, Basetrainable_parameters=3.6M, dimensionality=8, decoder=Base2022.09 | 119.1 | — | — | — | 0.367 | 0.278 | — | — | — | — | — | 0.211 | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP-2Evaluation Protocol=Zero-shot2023.12 | 118.8 | — | — | — | 0.356 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SMALLCAP d=4, Basetrainable_parameters=1.8M, dimensionality=4, decoder=Base2022.09 | 117.4 | — | — | — | 0.36 | 0.274 | — | — | — | — | — | 0.21 | — | — | — | — | — | — | — | — | — | — | — | |
| I-Tuning Basetrainable_parameters=14M2022.09 | 116.7 | — | — | — | 0.348 | 0.283 | — | — | — | — | — | 0.218 | — | — | — | — | — | — | — | — | — | — | — | |
| DeeBLIPArchitecture=Early Exit, Speed=1.65x2025.06 | 115.1 | — | — | — | 0.328 | 20.9 | — | — | — | — | — | 25.3 | — | — | — | — | — | — | — | — | — | — | — | |
| ClipCaptrainable_parameters=43M2022.09 | 113.1 | — | — | — | 0.335 | 0.275 | — | — | — | — | — | 0.211 | — | — | — | — | — | — | — | — | — | — | — | |
| Enc-DecPre-train #Images=15M2022.01 | 110.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PABEEBackbone=OFA2022.11 | 105.8 | — | — | — | 0.314 | 0.268 | — | — | — | — | — | 0.21 | — | — | — | — | — | — | — | -16.3 | — | — | — | |
| DeeBERTBackbone=OFA2022.11 | 102.1 | — | — | — | 0.301 | 0.263 | — | — | — | — | — | 0.205 | — | — | — | — | — | — | — | -15.5 | — | — | — | |
| L-versecaption_truncation=false2021.11 | 1.816 | — | — | — | 0.276 | 0.236 | — | — | — | — | 0.439 | 28.9 | — | — | — | — | — | — | — | — | — | — | — | |
| mPLUGData=14M, Optimization=CIDEr2022.05 | 1.551 | — | — | — | 46.5 | 32 | — | — | — | — | — | 26 | — | — | — | — | — | — | — | — | — | — | — | |
| OFAOptimization Objective=CIDEr Optimization2022.02 | 1.549 | — | — | — | 0.449 | 0.325 | — | — | — | — | — | 0.266 | — | — | — | — | — | — | — | — | — | — | — | |
| OFA LargeOptimization Objective=CIDEr Optimization2022.02 | 1.507 | — | — | — | 0.436 | 0.322 | — | — | — | — | — | 0.262 | — | — | — | — | — | — | — | — | — | — | — | |
| SMOLA-PALI-XFTCategory=Generalist, Fine-tuned=true2023.12 | 1.498 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OFAData=18M, Optimization=CIDEr2022.05 | 1.496 | — | — | — | 43.5 | 31.9 | — | — | — | — | — | 26.1 | — | — | — | — | — | — | — | — | — | — | — | |
| PaLI-XParameters=55B, Fine-tuning=per-task, OCR_pipeline=without2023.05 | 1.492 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaLI-XNumber of parameters=55B2023.05 | 1.492 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |