Video Captioning on MSVD
195.6CIDErMaMMUT
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| MaMMUTpre-training=image-language only, finetuning=direct2023.03 | 195.6 | — | — | — | — | — | |
| GIT2#PT Data=12.9B2023.02 | 185.4 | 82.2 | 52.3 | 88.7 | — | — | |
| GIT-2Vision Encoder=DaViT-4.8B, Image-Text Data=12.9B pairs2023.10 | 185.4 | 82.2 | 52.3 | — | — | 88.7 | |
| GIT2 (5.1B)#Example=12.9B, Mod=V2023.04 | 185.4 | 82.2 | 52.3 | 88.7 | — | — | |
| GIT22022.05 | 185.4 | 82.2 | 52.3 | — | — | 88.7 | |
| GIT22022.05 | 185.4 | 82.2 | 52.3 | 88.7 | — | — | |
| GIT22023.03 | 185.2 | — | — | — | — | — | |
| GIT#PT Data=0.8B2023.02 | 180.2 | 79.5 | 51.1 | 87.3 | — | — | |
| GIT2023.03 | 180.2 | — | — | — | — | — | |
| GIT2024.04 | 180.2 | — | 51.1 | — | — | — | |
| GIT#Example=800M, Mod=V2023.04 | 180.2 | 79.5 | 51.1 | 87.3 | — | — | |
| GIT2022.05 | 180.2 | 79.5 | 51.1 | — | — | 87.3 | |
| GIT2022.05 | 180.2 | 79.5 | 51.1 | 87.3 | — | — | |
| MA-LMM2024.04 | 179.1 | — | 51 | — | — | — | |
| VALORL#Example=33.5M, Mod=V+A2023.04 | 178.5 | 80.7 | 51 | 87.9 | — | — | |
| Video-LLaMA2024.04 | 175.3 | — | 49.8 | — | — | — | |
| mPLUG-2#PT Data=17M2023.02 | 165.8 | 75 | 48.4 | 85.3 | — | — | |
| mPLUG-22024.04 | 165.8 | — | 48.4 | — | — | — | |
| GITL#Example=20M, Mod=V2023.04 | 162.9 | 75.8 | 48.7 | 85.5 | — | — | |
| GIT-Large2022.05 | 162.9 | 75.8 | 48.7 | — | — | 85.5 | |
| GITLArchitecture=Large2022.05 | 162.9 | 75.8 | 48.7 | 85.5 | — | — | |
| VALORB#Example=6.5M, Mod=V+A2023.04 | 162.1 | 76.1 | 48 | 85.2 | — | — | |
| VALOR#Example=5.5M, Mod=V2023.04 | 156.1 | 74.3 | 47.1 | 83.8 | — | — | |
| HowToCaptionVision Encoder=ViT-B, Image-Text Data=6 datasets (CC3M+COCO+VG+SBU+CC12M+LAION), Video-Text Data=HowToCaption, Fine-tuned=true2023.10 | 154.2 | 70.4 | 46.4 | — | — | 83.2 | |
| BLIP + HowTo100MVision Encoder=ViT-B, Image-Text Data=6 datasets (CC3M+COCO+VG+SBU+CC12M+LAION), Video-Text Data=HowTo100M, Fine-tuned=true2023.10 | 151.4 | 69.3 | 46.2 | — | — | 83 | |
| LAVENDERVideo-Text Pre-training=true2023.01 | 150.7 | — | — | — | — | — | |
| LAVENDER#PT Data=30M2023.02 | 150.7 | — | — | — | — | — | |
| LAVENDERVision Encoder=ViViT-B, Video-Text Data=WebVid2M+12M2023.10 | 150.7 | — | — | — | — | — | |
| LAVENDER#Example=30M, Mod=V2023.04 | 150.7 | — | — | — | — | — | |
| BLIPVision Encoder=ViT-B, Image-Text Data=6 datasets (CC3M+COCO+VG+SBU+CC12M+LAION)2023.10 | 148.6 | 68 | 45.3 | — | — | 82.8 | |
| mPLUG-2Base#PT Data=17M2023.02 | 148.2 | 69.3 | 45.1 | 81.9 | — | — | |
| mPlug-2Vision Encoder=MViT-B, Image-Text Data=5 datasets (CC3M+COCO+VG+SBU+CC12M), Video-Text Data=WebVid2M2023.10 | 148.2 | 69.3 | 45.1 | — | — | 81.9 | |
| SeViT† FiDVideo-Text Pre-training=false, Initialization=OFA-Large backbone2023.01 | 148.1 | 69.1 | 46.3 | 83 | — | — | |
| HiTeA# PT Data=17M2022.12 | 146.9 | — | — | — | — | — | |
| HiTeA#PT Data=17M2023.02 | 146.9 | 71 | 45.3 | 81.4 | — | — | |
| HiTeAVision Encoder=VidSwin-B, Image-Text Data=5 datasets (CC3M+COCO+VG+SBU+CC12M), Video-Text Data=WebVid2M2023.10 | 146.9 | — | — | — | — | — | |
| Vid2Seq2023.03 | 146.2 | — | — | — | — | — | |
| Vid2SeqVision Encoder=ViT-L, Video-Text Data=YT-Temporal-1B2023.10 | 146.2 | 45.3 | — | — | — | — | |
| ClusterSTM#Pairs=5M2026.03 | 145.6 | — | — | — | — | — | |
| STM#Pairs=5M2026.03 | 145.4 | — | — | — | — | — | |
| HiTeA# PT Data=5M2022.12 | 145.1 | — | — | — | — | — | |
| HiTeA#Pairs=5M2026.03 | 145.1 | — | — | — | — | — | |
| LAVENDER# PT Data=5M2022.12 | 142.9 | — | — | — | — | — | |
| GIT-BVision Encoder=ViT-B, Image-Text Data=4 datasets (CC3M+COCO+VG+SBU)2023.10 | 142.6 | 69.3 | 44.5 | — | — | 81.4 | |
| GIT-Base2022.05 | 142.6 | 69.3 | 44.5 | — | — | 81.4 | |
| GITBArchitecture=Base2022.05 | 142.6 | 69.3 | 44.5 | 81.4 | — | — | |
| VIOLETV2# Pretrain videos/images=5M2022.09 | 139.2 | — | — | — | — | — | |
| VIOLETv2#Example=5.5M, Mod=V2023.04 | 139.2 | — | — | — | — | — | |
| VIOLETv2#Pairs=5M2026.03 | 139.2 | — | — | — | — | — | |
| LAVENDER#Pairs=5M2026.03 | 139.2 | — | — | — | — | — | |
| SeViT MARVideo-Text Pre-training=false2023.01 | 136.1 | 64.6 | 42.9 | 80.3 | — | — | |
| SeViT FiDVideo-Text Pre-training=false2023.01 | 135.5 | 66.9 | 43.7 | 80.9 | — | — | |
| SeViT-S FiDVideo-Text Pre-training=false, Variant=S (Small)2023.01 | 134.9 | 65.7 | 43.3 | 79.8 | — | — | |
| STOA-VLPPT=true2023.12 | 131.8 | 64.4 | — | — | — | 83.9 | |
| Uni-PerceiverTuning setting=Fine-tuning (FT), Data percentage=100%2021.12 | 131 | 61.5 | 42.3 | 79 | 7.7 | — | |
| VIOLETV22023.03 | 130.2 | — | — | — | — | — | |
| SeViT-S MARVideo-Text Pre-training=false, Variant=S (Small)2023.01 | 127.4 | 64 | 42.9 | 79.5 | — | — | |
| Tarsier-34BZero-shot=true2024.06 | 125.9 | — | — | — | — | — | |
| RTQPT=false2023.12 | 123.4 | 66.9 | — | — | — | 82.8 | |
| OmniViD2024.03 | 122.5 | 59.7 | 42.2 | 78.1 | — | — | |
| SwinBERT2022.12 | 120.6 | — | — | — | — | — | |
| SwinBERTVideo-Text Pre-training=false2023.01 | 120.6 | 58.2 | 41.3 | 77.5 | — | — | |
| SwinBERT2023.02 | 120.6 | 58.2 | 41.3 | 77.5 | — | — | |
| SwinBERT2022.09 | 120.6 | — | — | — | — | — | |
| SWINBert2023.03 | 120.6 | — | — | — | — | — | |
| SwinBERTPT=false2023.12 | 120.6 | 58.2 | — | — | — | 77.5 | |
| SwinBERT2024.04 | 120.6 | — | 41.3 | — | — | — | |
| SwinBERTVision Encoder=VidSwin-B2023.10 | 120.6 | 58.2 | 41.3 | — | — | 77.5 | |
| SwinBERTMod=V2023.04 | 120.6 | 58.2 | 41.3 | 77.5 | — | — | |
| SwinBERT2022.05 | 120.6 | 58.2 | 41.3 | — | — | 77.5 | |
| SwinBERT2022.05 | 120.6 | 58.2 | 41.3 | 77.5 | — | — | |
| SwinBERT2024.03 | 120.6 | 58.2 | 41.3 | 77.5 | — | — | |
| SwinBERT#Pairs=5M2026.03 | 120.6 | — | — | — | — | — | |
| Uni-PerceiverTuning setting=Prompt-tuning (PT), Data percentage=10%2021.12 | 112.1 | 57.2 | 39.1 | 75.6 | 6.8 | — | |
| KG-VCNVenue=PR’25, Method Type=Autoregressive2026.04 | 107.1 | 64.9 | 39.7 | — | — | 77.2 | |
| TextKGPT=false2023.12 | 105.2 | 60.8 | — | — | — | 75.1 | |
| Uni-PerceiverTuning setting=Prompt-tuning (PT), Data percentage=1%2021.12 | 104.8 | 54.8 | 38.9 | 74.7 | 6.6 | — | |
| HMNPT=false2023.12 | 104 | 59.2 | — | — | — | 75.1 | |
| MUFIBackbone=ResNet-502022.01 | 101.5 | 56.9 | 37.4 | 74.2 | — | — | |
| MGRMPVenue=ICCV’21, Method Type=Autoregressive2026.04 | 98.5 | 55.8 | 36.9 | — | — | 74.5 | |
| LSRTVenue=TIP’22, Method Type=Autoregressive2026.04 | 98.5 | 55.6 | 37.1 | — | — | 73.5 | |
| EFFECTVenue=TOMM’23, Method Type=Autoregressive2026.04 | 98.5 | 56.9 | 36.6 | — | — | 74.2 | |
| Tarsier-7BZero-shot=true2024.06 | 98 | — | — | — | — | — | |
| RSFDVenue=AAAI’23, Method Type=Autoregressive2026.04 | 96.7 | 51.2 | 35.7 | — | — | 72.9 | |
| DiffVCVenue=-, Method Type=Non-autoregressive2026.04 | 95.6 | 53.5 | 37.1 | — | — | 72.5 | |
| ORG-TRL2021.12 | 95.2 | 54.3 | 36.4 | 73.9 | — | — | |
| ORG-TRLBackbone=IncResV2+C3D+FasterRCNN2022.01 | 95.2 | 54.3 | 36.4 | 73.9 | — | — | |
| ORG-TRLVideo-Text Pre-training=false2023.01 | 95.2 | 54.3 | 36.4 | 73.9 | — | — | |
| ORG-TRL2023.03 | 95.2 | — | — | — | — | — | |
| ORG-TRLMod=V2023.04 | 95.2 | 54.3 | 36.4 | 73.9 | — | — | |
| ORG-TRL2022.05 | 95.2 | 54.3 | 36.4 | — | — | 73.9 | |
| ORG-TRL2022.05 | 95.2 | 54.3 | 36.4 | 73.9 | — | — | |
| ORG-TRLOff-the-shelf object detectors=true2024.03 | 95.2 | 54.3 | 36.4 | 73.9 | — | — | |
| ORG-TRLVenue=CVPR’20, Method Type=Autoregressive2026.04 | 95.2 | 54.3 | 36.4 | — | — | 73.9 | |
| PMI-CAP2022.05 | 95.1 | 54.6 | 36.4 | — | — | — | |
| PMI-CAP2022.05 | 95.1 | 54.6 | 36.4 | — | — | — | |
| PMI-CAP2024.03 | 95.1 | 54.6 | 36.4 | — | — | — | |
| PMI-CAPVenue=ECCV’20, Method Type=Autoregressive2026.04 | 95.1 | 54.6 | 36.4 | — | — | — | |
| SGNVenue=AAAI’21, Method Type=Autoregressive2026.04 | 94.3 | 52.8 | 35.5 | — | — | 72.9 | |
| InternVideo2Zero-shot=true2024.06 | 93.1 | — | — | — | — | — |