Video Captioning on VATEX (test)
84.2CIDErFlamingo
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Flamingomode=Fine-tuned2022.04 | 84.2 | — | — | — | — | — | — | — | — | — | — | — | |
| FlamingoEvaluation Protocol=Fine-tuned2022.04 | 84.2 | — | — | — | — | — | — | — | — | — | — | — | |
| SotAoptimization=domain-specific metric2022.04 | 81.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Unrestricted SotA (VIOLET)Evaluation Protocol=Fine-tuned2022.04 | 81.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Restricted SotA+ (VIOLET)Evaluation Protocol=Fine-tuned2022.04 | 76.3 | — | — | — | — | — | — | — | — | — | — | — | |
| SigLIP-so400m/14LLM=Fine-tuned2026.07 | 66 | — | — | — | — | 34 | — | — | — | — | — | — | |
| FlamingoFew-shot=32 shots2022.04 | 65.1 | — | — | — | — | — | — | — | — | — | — | — | |
| FlamingoShots=32, Evaluation Protocol=In-context learning2022.04 | 65.1 | — | — | — | — | — | — | — | — | — | — | — | |
| SigLIP-so400m/14LLM=Frozen2026.07 | 64.5 | — | — | — | — | 32.9 | — | — | — | — | — | — | |
| Flamingo-80B#VideoPT=27M, ASR=No, Evaluation Protocol=Few-shot, Num Shots=162022.05 | 62.8 | — | — | — | — | — | — | — | — | — | — | — | |
| CLIP-DCDTraining Mode=Ours2021.11 | 62.4 | — | — | — | — | 36.8 | 25.1 | 52.2 | — | — | — | — | |
| CLIP-BaseTraining Mode=Ours2021.11 | 60.9 | — | — | — | — | 36.8 | 24.8 | 51.9 | — | — | — | — | |
| L_ITG + L_alignmenttraining=end-to-end with ITG + Alignment Loss, backbone=I3D -> Transformer -> OPT2.7B2023.07 | 60.9 | — | — | — | — | 30.9 | — | — | — | 49.1 | — | — | |
| SigLIP2-B/16LLM=Fine-tuned2026.07 | 58.4 | — | — | — | — | 31.4 | — | — | — | — | — | — | |
| MGCMPYear=20212021.11 | 57.6 | — | — | — | — | 34.2 | 23.5 | 50.3 | — | — | — | — | |
| OpenBookYear=20212021.11 | 57.5 | — | — | — | — | 33.9 | 23.7 | 50.2 | — | — | — | — | |
| NSAYear=20202021.11 | 57.1 | — | — | — | — | 31 | 22.7 | 49 | — | — | — | — | |
| Flamingo-3B#VideoPT=27M, ASR=No, Evaluation Protocol=Few-shot, Num Shots=162022.05 | 57.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Gen4U @ 30%Noise Aug.=true, LLM=Fine-tuned2026.07 | 56.7 | — | — | — | — | 32.3 | — | — | — | — | — | — | |
| L_ITGtraining=end-to-end with ITG, backbone=I3D -> Transformer -> OPT2.7B2023.07 | 56.6 | — | — | — | — | 29.3 | — | — | — | 48.2 | — | — | |
| SigLIP2-B/16LLM=Frozen2026.07 | 56.2 | — | — | — | — | 30.5 | — | — | — | — | — | — | |
| ORG-TRLYear=20202021.11 | 49.7 | — | — | — | — | 32.1 | 22.2 | 48.9 | — | — | — | — | |
| ORG-TRL2023.07 | 49.7 | — | — | — | — | 32.1 | — | — | — | 48.9 | — | — | |
| Gen4U @ 30%Noise Aug.=true, LLM=Frozen2026.07 | 48.5 | — | — | — | — | 29.6 | — | — | — | — | — | — | |
| VATEXYear=20192021.11 | 45.1 | — | — | — | — | 28.4 | 21.7 | 47 | — | — | — | — | |
| Gen4U @ 30%LLM=Fine-tuned2026.07 | 44.8 | — | — | — | — | 28.1 | — | — | — | — | — | — | |
| ABMambaSize=3.6B, Throughput=83.8, Category=Ours2026.04 | 44.4 | — | — | — | — | 28.6 | 22.2 | — | — | 47.7 | 73.4 | 41.8 | |
| mPLUGEvaluation protocol=Zero-Shot2022.05 | 42 | — | — | — | — | — | — | — | — | — | — | — | |
| Gen4U @ 30%LLM=Frozen2026.07 | 40.2 | — | — | — | — | 26.5 | — | — | — | — | — | — | |
| Flamingo-3BTraining data=27M VT & 2.1B IT & 43M WP, Pre-training strategy=Web video-text Pre-training, Zero-shot=true2024.05 | 40.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Flamingo-9BTraining data=27M VT & 2.1B IT & 43M WP, Pre-training strategy=Web video-text Pre-training, Zero-shot=true2024.05 | 39.5 | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-OneVisionSize=7B, Throughput=17.2, Category=Fully open MLLMs2026.04 | 39.2 | — | — | — | — | 17.6 | 21.2 | — | — | 41.5 | 60.6 | 42 | |
| VidIL#VideoPT=0, ASR=Yes, Evaluation Protocol=Few-shot, Num Shots=102022.05 | 38.9 | — | — | — | — | 23.2 | 20.6 | 44.2 | 75.3 | — | — | — | |
| Video-LLaVASize=7B, Throughput=28.4, Category=Fully open MLLMs2026.04 | 37.7 | — | — | — | — | 24.5 | 19.6 | — | — | 44.9 | 67.3 | 40.8 | |
| BLIPEvaluation protocol=Zero-Shot2022.05 | 37.4 | — | — | — | — | — | — | — | — | — | — | — | |
| VideoLLaMA3Size=2B, Throughput=29.7, Category=Small MLLMs2026.04 | 36.9 | — | — | — | — | 17.5 | 24 | — | — | 42.2 | 56.8 | 43 | |
| VidIL#VideoPT=0, ASR=No, Evaluation Protocol=Few-shot, Num Shots=102022.05 | 36.7 | — | — | — | — | 22.2 | 20 | 43.6 | 33.3 | — | — | — | |
| UniVL#VideoPT=1.2M, ASR=Yes, Evaluation Protocol=Fine-tuning2022.05 | 35.6 | — | — | — | — | 23.7 | 22.7 | 39.3 | 106.2 | — | — | — | |
| UniVL#VideoPT=1.2M, ASR=No, Evaluation Protocol=Fine-tuning2022.05 | 33.4 | — | — | — | — | 22.8 | 22.3 | 38.6 | 70.2 | — | — | — | |
| InternVL2.5Size=2.2B, Throughput=29.3, Category=Small MLLMs2026.04 | 33.2 | — | — | — | — | 16.5 | 17.2 | — | — | 35.8 | 58.3 | 41.4 | |
| TOPA-Llama2-13BTraining data=721K TextVid, Pre-training strategy=Text-only Pre-training, Zero-shot=true2024.05 | 32 | — | — | — | — | — | — | — | — | — | — | — | |
| VideoPrism-B w/ PaLM-2-8BTraining data=618M VT, Backbone=PaLM-2-8B, Pre-training strategy=Web video-text Pre-training, Zero-shot=true2024.05 | 31.7 | — | — | — | — | — | — | — | — | — | — | — | |
| TOPA-Llama2-7BTraining data=721K TextVid, Pre-training strategy=Text-only Pre-training, Zero-shot=true2024.05 | 31 | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP_cap#VideoPT=0, ASR=No, Evaluation Protocol=Few-shot, Num Shots=102022.05 | 28.9 | — | — | — | — | 20.7 | 17.4 | 41.5 | 22.8 | — | — | — | |
| VideoPrism-B w/ PaLM-2-1BTraining data=618M VT, Backbone=PaLM-2-1B, Pre-training strategy=Web video-text Pre-training, Zero-shot=true2024.05 | 24.2 | — | — | — | — | — | — | — | — | — | — | — | |
| NITS-VC2023.07 | 24 | — | — | — | — | 20 | — | — | — | 42 | — | — | |
| VideoCoCa-gTraining data=144M VT, Pre-training strategy=Web video-text Pre-training, Zero-shot=true2024.05 | 22.8 | — | — | — | — | — | — | — | — | — | — | — | |
| BLIP#VideoPT=0, ASR=No, Evaluation Protocol=Few-shot, Num Shots=102022.05 | 20.7 | — | — | — | — | 13.5 | 15.4 | 39.5 | 23.9 | — | — | — | |
| DeCapTraining data=3M Captions, Pre-training strategy=Text-only Pre-training, Zero-shot=true2024.05 | 18.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Gemini-1.5-ProSize=–, Category=Proprietary2026.04 | 14.2 | — | — | — | — | 4.5 | 8.9 | — | — | 19.3 | 22.8 | 39.2 | |
| Video-ChatGPTSize=7B, Throughput=35.2, Category=Fully open MLLMs2026.04 | 9.9 | — | — | — | — | 2.4 | 10.6 | — | — | 25 | 14.8 | 39.4 | |
| UniVL#VideoPT=1.2M, ASR=Yes, Evaluation Protocol=Few-shot, Num Shots=102022.05 | 3.4 | — | — | — | — | 2.7 | 10.2 | 17.7 | 26 | — | — | — | |
| UniVL#VideoPT=1.2M, ASR=No, Evaluation Protocol=Few-shot, Num Shots=102022.05 | 2.1 | — | — | — | — | 1.7 | 8 | 15.7 | 13.3 | — | — | — | |
| COSA (1.2B)Parameters=415M, SCST finetuning=true2023.06 | 0.965 | — | — | — | — | 0.437 | — | — | — | — | — | — | |
| VALOR-LParameters=433.5M, SCST finetuning=true, Additional modalities=true2023.06 | 0.958 | — | — | — | — | 0.456 | — | — | — | — | — | — | |
| GIT2Parameters=12.9B, SCST finetuning=true2023.06 | 0.945 | — | — | — | — | 0.427 | — | — | — | — | — | — | |
| GITParameters=1.7B, SCST finetuning=true2023.06 | 0.915 | — | — | — | — | 0.416 | — | — | — | — | — | — | |
| COSA-LParameters=417M2023.06 | 0.844 | — | — | — | — | 0.454 | — | — | — | — | — | — | |
| VideoCoCaPre-trained on large-scale datasets=true2023.09 | 0.778 | — | — | — | — | 0.397 | — | — | — | 0.545 | — | — | |
| COSA-BParameters=17M2023.06 | 0.744 | — | — | — | — | 0.414 | — | — | — | — | — | — | |
| SwinBERTPre-trained on large-scale datasets=true2023.09 | 0.73 | — | — | — | — | 0.387 | 0.262 | — | — | 0.532 | — | — | |
| COSA-BParameters=5M2023.06 | 0.726 | — | — | — | — | 0.405 | — | — | — | — | — | — | |
| CoCapPre-trained on large-scale datasets=true, Backbone=ViT/L142023.09 | 0.648 | — | — | — | — | 0.358 | 0.253 | — | — | 0.52 | — | — | |
| CoCapPre-trained on large-scale datasets=true, Backbone=B/162023.09 | 0.527 | — | — | — | — | 0.314 | 0.232 | — | — | 0.494 | — | — | |
| Support-setPre-trained on large-scale datasets=false2023.09 | 0.512 | — | — | — | — | 0.328 | 0.244 | — | — | 0.491 | — | — | |
| ORG-TRLPre-trained on large-scale datasets=false2023.09 | 0.497 | — | — | — | — | 0.321 | 0.222 | — | — | 0.489 | — | — | |
| VATEXPre-trained on large-scale datasets=false2023.09 | 0.451 | — | — | — | — | 0.284 | 0.217 | — | — | 0.47 | — | — | |
| NITS-VCPre-trained on large-scale datasets=false2023.09 | 0.24 | — | — | — | — | 0.2 | 0.18 | — | — | 0.42 | — | — | |
| ORG-TRL2021.11 | 0.185 | 0.751 | 0.539 | 0.606 | 0.527 | — | — | — | — | — | — | — | |
| GT2021.11 | 0.178 | 0.937 | 0.581 | 0.639 | 0.498 | — | — | — | — | — | — | — | |
| Top-Down2021.11 | 0.173 | 0.73 | 0.53 | 0.591 | 0.515 | — | — | — | — | — | — | — | |
| AM_12021.11 | 0.146 | 0.729 | 0.522 | 0.584 | 0.464 | — | — | — | — | — | — | — | |
| AM_22021.11 | 0.14 | 0.714 | 0.515 | 0.571 | 0.451 | — | — | — | — | — | — | — | |
| AM_32021.11 | 0.134 | 0.698 | 0.512 | 0.566 | 0.447 | — | — | — | — | — | — | — |