Video Captioning on MSRVTT (B@4, M, R, C)
80.3CIDErmPLUG-2
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| mPLUG-2#PT Data=17M2023.02 | 80.3 | 57.8 | 34.9 | — | 70.1 | |
| mPLUG-22023.05 | 80.3 | 57.8 | 34.9 | — | — | |
| mPLUG-22024.04 | 80.3 | — | 34.9 | — | — | |
| VAST(1.3B)Sample=442M, Audio or subtitle modalities=true2023.05 | 78 | 56.7 | — | — | — | |
| GIT2#PT Data=12.9B2023.02 | 75.9 | 54.8 | 33.1 | — | 68.2 | |
| GIT2023.05 | 75.9 | 54.8 | 33.1 | — | — | |
| GIT2(5.1B)Sample=12.9B, Audio or subtitle modalities=false2023.05 | 75.9 | 54.8 | — | — | — | |
| GIT-2Vision Encoder=DaViT-4.8B, Image-Text Data=12.9B pairs2023.10 | 75.9 | 54.8 | 33.1 | — | 68.2 | |
| GIT2 (5.1B)#Example=12.9B, Mod=V2023.04 | 75.9 | 54.8 | 33.1 | 68.2 | — | |
| GIT22022.05 | 75.9 | 54.8 | 33.1 | 68.2 | — | |
| MA-LMM2024.04 | 74.6 | — | 33.4 | — | — | |
| CoDi2023.05 | 74.4 | 52.1 | 32.5 | — | — | |
| VALOR-LSample=433.5M, Audio or subtitle modalities=true2023.05 | 74 | 54.4 | — | — | — | |
| VALORL#Example=33.5M, Mod=V+A2023.04 | 74 | 54.4 | 32.9 | 68 | — | |
| GIT#PT Data=0.8B2023.02 | 73.9 | 53.8 | 32.9 | — | 67.7 | |
| GITSample=1.7B, Audio or subtitle modalities=false2023.05 | 73.9 | 53.8 | — | — | — | |
| GIT2024.04 | 73.9 | — | 32.9 | — | — | |
| GIT#Example=800M, Mod=V2023.04 | 73.9 | 53.8 | 32.9 | 67.7 | — | |
| GIT2022.05 | 73.9 | 53.8 | 32.9 | 67.7 | — | |
| MaMMUTSample=2B, Audio or subtitle modalities=false2023.05 | 73.6 | — | — | — | — | |
| VideoCoca#PT Data=3B2023.02 | 73.2 | 53.8 | — | — | 68 | |
| VideoCoCa2024.04 | 73.2 | — | — | — | — | |
| mPLUG-2Base#PT Data=17M2023.02 | 72.4 | 52.2 | 32.1 | — | 66.9 | |
| mPlug-2Vision Encoder=MViT-B, Image-Text Data=5 datasets (CC3M+COCO+VG+SBU+CC12M), Video-Text Data=WebVid2M2023.10 | 72.4 | 52.2 | 32.1 | — | 66.9 | |
| Video-LLaMA2024.04 | 71.6 | — | 32.9 | — | — | |
| RTQPT=false2023.12 | 69.3 | 49.6 | — | — | 66.1 | |
| VideoCoCaVision Encoder=ViT-G, Image-Text Data=JFT-3B, Video-Text Data=VideoCC3M2023.10 | 68 | 49.8 | — | — | — | |
| VALORB#Example=6.5M, Mod=V+A2023.04 | 66.6 | 53.8 | 32.3 | 67 | — | |
| BLIP + HowTo100MVision Encoder=ViT-B, Image-Text Data=6 datasets (CC3M+COCO+VG+SBU+CC12M+LAION), Video-Text Data=HowTo100M, Fine-tuned=true2023.10 | 65.9 | 49.4 | 32 | — | 66.2 | |
| BLIPVision Encoder=ViT-B, Image-Text Data=6 datasets (CC3M+COCO+VG+SBU+CC12M+LAION)2023.10 | 65.5 | 49.2 | 32.2 | — | 66.1 | |
| HowToCaptionVision Encoder=ViT-B, Image-Text Data=6 datasets (CC3M+COCO+VG+SBU+CC12M+LAION), Video-Text Data=HowToCaption, Fine-tuned=true2023.10 | 65.3 | 49.8 | 32.2 | — | 66.3 | |
| HiTeA#PT Data=17M2023.02 | 65.1 | 49.2 | 30.7 | — | 65 | |
| Vid2SeqVision Encoder=ViT-L, Video-Text Data=YT-Temporal-1B2023.10 | 64.6 | 50.8 | — | — | — | |
| GITL#Example=20M, Mod=V2023.04 | 64.1 | 48.7 | 30.9 | 64.9 | — | |
| GIT-Large2022.05 | 64.1 | 48.7 | 30.9 | 64.9 | — | |
| HiTeASample=5M, Audio or subtitle modalities=false2023.05 | 62.5 | — | — | — | — | |
| VALOR#Example=5.5M, Mod=V2023.04 | 61.5 | 48.1 | 30.4 | 64.3 | — | |
| TextKG (CLIP)Input=V+S, Features=CLIP2023.03 | 60.8 | 46.6 | 30.5 | — | 64.8 | |
| TextKGPT=false2023.12 | 60.8 | 46.6 | — | — | 64.8 | |
| STOA-VLPPT=true2023.12 | 60.2 | 45.8 | — | — | 68.4 | |
| LAVENDERInput=V, Features=VidSwin2023.03 | 60.1 | — | — | — | — | |
| LAVENDERSample=30M, Audio or subtitle modalities=false2023.05 | 60.1 | — | — | — | — | |
| LAVENDER#Example=30M, Mod=V2023.04 | 60.1 | — | — | — | — | |
| MV-GPTInput=V+S, Features=ViViT2023.03 | 60 | 48.9 | 38.7 | — | 64 | |
| MV-GPT#PT Data=69M2023.02 | 60 | 48.9 | 38.7 | — | 64 | |
| MV-GPT2023.05 | 60 | 48.9 | 38.7 | — | — | |
| MV-GPTPT=true2023.12 | 60 | 48.9 | — | — | 64 | |
| MV-GPTVision Encoder=ViT-B, Image-Text Data=4 datasets (CC3M+COCO+VG+SBU), Video-Text Data=HowTo100M2023.10 | 60 | 48.9 | 38.6 | — | 64 | |
| MV-GPT#Example=136M, Mod=V+S2023.04 | 60 | 48.9 | 38.7 | 64 | — | |
| MV-GPTwith subtitle=true2022.05 | 60 | 48.9 | 38.7 | 64 | — | |
| CMVCInput=V, Features=CLIP2023.03 | 58.7 | 48.2 | 31.3 | — | 64.8 | |
| CMVCPT=false2023.12 | 58.7 | 48.2 | — | — | 64.8 | |
| SMPFFSample=None, Audio or subtitle modalities=true2023.05 | 58.5 | 48.4 | — | — | — | |
| SMPFFMod=V+A2023.04 | 58.5 | 48.4 | 30.6 | 64.9 | — | |
| VIOLETv2Sample=5M, Audio or subtitle modalities=false2023.05 | 58 | — | — | — | — | |
| VIOLETv2#Example=5.5M, Mod=V2023.04 | 58 | — | — | — | — | |
| GIT-BVision Encoder=ViT-B, Image-Text Data=4 datasets (CC3M+COCO+VG+SBU)2023.10 | 57.8 | 46.6 | 29.6 | — | 63.2 | |
| GIT-Base2022.05 | 57.8 | 46.6 | 29.6 | 63.2 | — | |
| CLIP4CInput=V, Features=CLIP2023.03 | 57.7 | 46.1 | 30.7 | — | 63.7 | |
| CLIP4Caption2023.02 | 57.7 | 46.1 | 30.7 | — | 63.7 | |
| Clip4CapPT=false2023.12 | 57.7 | 46.1 | — | — | 63.7 | |
| CLIP4CaptionVision Encoder=ViT-B2023.10 | 57.7 | 46.1 | 30.7 | — | 63.7 | |
| OmniViD2024.03 | 56.6 | 44.3 | 29.9 | 62.7 | — | |
| SWINBERTInput=V, Features=VidSwin2023.03 | 55.9 | 45.4 | 30.6 | — | 64.1 | |
| SwinBERTPT=false2023.12 | 55.9 | 45.4 | — | — | 64.1 | |
| SwinBERT2023.02 | 53.8 | 41.9 | 29.9 | — | 62.1 | |
| SwinBERTSample=None, Audio or subtitle modalities=false2023.05 | 53.8 | 41.9 | — | — | — | |
| SwinBERT2024.04 | 53.8 | — | 29.9 | — | — | |
| SwinBERTVision Encoder=VidSwin-B2023.10 | 53.8 | 41.9 | 29.9 | — | 62.1 | |
| SwinBERTMod=V2023.04 | 53.8 | 41.9 | 29.9 | 62.1 | — | |
| SwinBERT2022.05 | 53.8 | 41.9 | 29.9 | 62.1 | — | |
| SwinBERT2024.03 | 53.8 | 41.9 | 29.9 | 62.1 | — | |
| OpenBookMod=V2023.04 | 52.9 | 42.8 | 29.3 | 61.7 | — | |
| OpenBook2022.05 | 52.9 | 33.9 | 23.7 | 50.2 | — | |
| OpenBook2024.03 | 52.9 | 42.8 | 29.3 | 61.7 | — | |
| MELTRSample=136M, Audio or subtitle modalities=true2023.05 | 52.8 | 44.2 | — | — | — | |
| DeCEMInput=V+S, Features=BERT2023.03 | 52.3 | 45.2 | 29.7 | — | 64.7 | |
| HMNPT=false2023.12 | 51.5 | 43.5 | — | — | 62.7 | |
| SAAT2024.03 | 51 | 39.9 | 27.7 | 61.2 | — | |
| ORG-TRL2023.05 | 50.9 | 43.6 | 28.8 | — | — | |
| ORG-TRLMod=V2023.04 | 50.9 | 43.6 | 28.8 | 62.1 | — | |
| ORG-TRL2022.05 | 50.9 | 43.6 | 28.8 | 62.1 | — | |
| ORG-TRLOff-the-shelf object detectors=true2024.03 | 50.9 | 43.6 | 28.8 | 62.1 | — | |
| UniVLInput=V+S, Features=S3D2023.03 | 50 | 41.8 | 28.9 | — | 60.8 | |
| UniVLSample=136M, Audio or subtitle modalities=true2023.05 | 50 | 41.8 | — | — | — | |
| UniVLPT=true2023.12 | 50 | 41.8 | — | — | 60.8 | |
| UniVL#PT Data=136M2023.02 | 49.9 | 42.2 | 28.2 | — | 61.2 | |
| UniVL2024.04 | 49.9 | — | 28.2 | — | — | |
| PMI-CAP2022.05 | 49.4 | 42.1 | 28.7 | — | — | |
| PMI-CAP2024.03 | 49.4 | 42.1 | 28.7 | — | — | |
| POS+VCT2024.03 | 49.1 | 42.3 | 29.7 | 62.8 | — | |
| POS+CG2024.03 | 48.7 | 42 | 28.2 | 61.6 | — | |
| Support-set#Example=136M, Mod=V2023.04 | 48.6 | 38.9 | 28.2 | 59.8 | — | |
| Support-set2022.05 | 48.6 | 38.9 | 28.2 | 59.8 | — | |
| GRU-EVEOff-the-shelf object detectors=true2024.03 | 48.1 | 38.3 | 28.4 | 60.7 | — | |
| SibNet2024.03 | 47.5 | 40.9 | 27.5 | 60.2 | — | |
| MGSA2024.03 | 47.5 | 42.4 | 27.6 | — | — | |
| STG-KD2022.05 | 47.1 | 40.5 | 28.3 | 60.9 | — | |
| STG-KDOff-the-shelf object detectors=true2024.03 | 47.1 | 40.5 | 28.3 | 60.9 | — | |
| OA-BTGOff-the-shelf object detectors=true2024.03 | 46.9 | 41.4 | 28.2 | — | — |