Text-To-Video retrieval on MSRVTT (test)
46Recall@5CE
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| CEProtocol=Fine-Tuned, Note=use extra labeled data in the form of pre-trained semantic embeddings2020.03 | 46 | 18.2 | 60.7 | 7 | — | |
| Soft Max MarginProtocol=Fine-Tuned2020.03 | 41.6 | 17.4 | 53.6 | 8 | — | |
| HTM-PT*Protocol=Fine-Tuned2020.03 | 40.2 | 14.9 | 52.8 | 9 | — | |
| MoEEProtocol=Fine-Tuned2020.03 | 39.2 | 14.2 | 53.8 | 9 | — | |
| HTM-no-PTProtocol=Fine-Tuned2020.03 | 36 | 12.4 | 52 | 10 | — | |
| JSFusionProtocol=Fine-Tuned2020.03 | 31.2 | 10.2 | 43.2 | 13 | — | |
| MIL-NCEProtocol=Zero-Shot, Note=use additional clip-caption pairs2020.03 | 24 | 9.9 | 32.4 | 29.5 | — | |
| Soft Max MarginProtocol=Zero-Shot2020.03 | 21.3 | 8 | 29.3 | 33 | — | |
| HTM-PT*Protocol=Zero-Shot2020.03 | 21.2 | 7.5 | 29.6 | 38 | — | |
| JEMCProtocol=Fine-Tuned2020.03 | 20.9 | 7 | 29.7 | 38 | — | |
| VALOR_L#Example=33.5M, Mod=V+A, Dual Softmax (+DSL)=true2023.04 | 0.835 | 0.599 | 0.896 | — | — | |
| UMT-LExample=425M2023.06 | 0.81 | 0.588 | 0.871 | — | — | |
| CLIP-VIP#Example=100M, Mod=V, Dual Softmax (+DSL)=true2023.04 | 0.805 | 0.577 | 0.882 | — | — | |
| HunYuan_tvrMod=V, Dual Softmax (+DSL)=true2023.04 | 0.804 | 0.55 | 0.868 | — | — | |
| VALOR-LExample=433.5M2023.06 | 0.798 | 0.544 | 0.876 | — | — | |
| VALOR_L#Example=33.5M, Mod=V+A, Dual Softmax (+DSL)=false2023.04 | 0.798 | 0.544 | 0.876 | — | — | |
| COSAExample=415M2023.06 | 0.796 | 0.579 | 0.861 | — | — | |
| mPLUG-2Example=417M2023.06 | 0.776 | 0.531 | 0.847 | — | — | |
| CLIP-VIPExample=500M2023.06 | 0.772 | 0.542 | 0.848 | — | — | |
| COSA-LExample=417M2023.06 | 0.772 | 0.544 | 0.847 | — | — | |
| DCRMod=V2023.04 | 0.765 | 0.502 | 0.847 | — | — | |
| X-CLIPExample=400M2023.06 | 0.758 | 0.493 | 0.848 | — | — | |
| X-CLIPMod=V2023.04 | 0.758 | 0.493 | 0.848 | — | — | |
| TS2-NetMod=V2023.04 | 0.756 | 0.494 | 0.853 | — | — | |
| DiffusionRet2023.03 | 0.752 | 0.49 | 0.827 | 2 | 206.9 | |
| OmniVLExample=17M2023.06 | 0.742 | 0.478 | 0.838 | — | — | |
| X-CLIP# PT Data=400M, Inference Setting=Fine-tuning2022.12 | 0.73 | 0.461 | 0.831 | — | — | |
| EMCL-Net2023.03 | 0.723 | 0.47 | 0.826 | 2 | 201.9 | |
| VALOR_B#Example=6.5M, Mod=V+A2023.04 | 0.722 | 0.43 | 0.821 | — | — | |
| COSA-BExample=17M2023.06 | 0.721 | 0.469 | 0.813 | — | — | |
| Clip4Clip# PT Data=400M, Inference Setting=Fine-tuning2022.12 | 0.719 | 0.421 | 0.814 | — | — | |
| CLIP4ClipExample=400M2023.06 | 0.714 | 0.445 | 0.816 | — | — | |
| CLIP4ClipMod=V2023.04 | 0.714 | 0.445 | 0.816 | — | — | |
| HiTeA# PT Data=17M, Inference Setting=Fine-tuning2022.12 | 0.712 | 0.468 | 0.819 | — | — | |
| HiTeAExample=17M2023.06 | 0.712 | 0.468 | 0.819 | — | — | |
| CLIP4Clip2023.03 | 0.706 | 0.438 | 0.814 | 2 | 195.8 | |
| VINDLU-BExample=17M2023.06 | 0.699 | 0.453 | 0.796 | — | — | |
| CloverPre-training dataset=W2M+C3M, Evaluation Protocol=Fine-tune2022.07 | 0.698 | 0.405 | 0.794 | 2 | — | |
| mPLUG-2Zero-shot=true, Pre-training Data=17M2023.02 | 0.697 | 0.471 | 0.79 | — | — | |
| SINGULARITY-temporal#PT=17M, #Train Frame=42022.06 | 0.695 | 0.427 | 0.781 | — | — | |
| HiTeA# PT Data=5M, Inference Setting=Fine-tuning2022.12 | 0.693 | 0.444 | 0.789 | — | — | |
| COSA-BExample=5M2023.06 | 0.69 | 0.422 | 0.79 | — | — | |
| SINGULARITY#PT=17M, #Train Frame=12022.06 | 0.687 | 0.415 | 0.77 | — | — | |
| SINGULARITYExample=17M2023.06 | 0.687 | 0.415 | 0.77 | — | — | |
| SINGULARITY#Example=17M, Mod=V2023.04 | 0.687 | 0.415 | 0.77 | — | — | |
| CLIP4Clip#PT=400M, #Train Frame=12/64/642022.06 | 0.686 | 0.42 | 0.787 | — | — | |
| VASTSample=443M, Zero-shot=true, Modality=Omni-modal (Vision/Audio/Subtitle)2023.05 | 0.683 | 0.493 | 0.739 | — | — | |
| VASTDSize=154M, Params=1.3B, Zero-shot=true2024.01 | 0.683 | — | 0.739 | — | — | |
| All-in-1Pre-training dataset=W2M+H100M, Evaluation Protocol=Fine-tune2022.07 | 0.681 | 0.379 | 0.771 | — | — | |
| All-in-one# PT Data=138M, Inference Setting=Fine-tuning2022.12 | 0.681 | 0.379 | 0.771 | — | — | |
| All-in-oneExample=138M2023.06 | 0.681 | 0.379 | 0.771 | — | — | |
| All-in-one#Example=138M, Mod=V2023.04 | 0.681 | 0.379 | 0.771 | — | — | |
| CloverExample=5M2023.06 | 0.674 | 0.386 | 0.764 | — | — | |
| SINGULARITY-temporal#PT=5M, #Train Frame=42022.06 | 0.673 | 0.399 | 0.76 | — | — | |
| LAVENDERExample=30M2023.06 | 0.669 | 0.407 | 0.776 | — | — | |
| LAVENDER#Example=30M, Mod=V2023.04 | 0.669 | 0.407 | 0.776 | — | — | |
| SINGULARITY#PT=5M, #Train Frame=12022.06 | 0.659 | 0.368 | 0.755 | — | — | |
| Singularity# PT Data=5M, Inference Setting=Fine-tuning2022.12 | 0.659 | 0.368 | 0.755 | — | — | |
| BLIP#PT=130M, #Train Frame=1, Zero-shot=true2022.06 | 0.656 | 0.433 | 0.747 | — | — | |
| MV-GPT#Example=53M, Mod=V+S2023.04 | 0.655 | 0.373 | 0.751 | — | — | |
| HD-VILAPre-training dataset=HDV100M, Evaluation Protocol=Fine-tune2022.07 | 0.653 | 0.356 | 0.78 | 3 | — | |
| InternVideovisual_backbone=ViT, pretraining_pairs=640M, zero-shot=true2024.03 | 0.653 | 0.4 | 0.741 | — | — | |
| Nagrani et al.#Example=1.03M, Mod=V+A2023.04 | 0.651 | 0.358 | 0.769 | — | — | |
| SRL-CLIPBackbone=ViT-L/14, DSize=23k, Params=400M, Zero-shot=true2024.01 | 0.649 | — | 0.736 | 3 | — | |
| MCQPre-training dataset=W2M+C3M, Evaluation Protocol=Fine-tune2022.07 | 0.648 | 0.376 | 0.751 | 3 | — | |
| BridgeFormer# PT Data=5M, Inference Setting=Fine-tuning2022.12 | 0.648 | 0.376 | 0.751 | — | — | |
| BridgeFormerExample=5M2023.06 | 0.648 | 0.376 | 0.751 | — | — | |
| VIOLETv2Example=5M2023.06 | 0.648 | 0.372 | 0.758 | — | — | |
| BridgeFormer#Example=5.5M, Mod=V2023.04 | 0.648 | 0.376 | 0.751 | — | — | |
| VALOR_E#Example=5.5M, Mod=V2023.04 | 0.647 | 0.362 | 0.754 | — | — | |
| BT-AdapterDSize=2M, Params=450M, Zero-shot=true2024.01 | 0.647 | — | 0.735 | — | — | |
| LF-VILA#Example=8.5M, Mod=V2023.04 | 0.645 | 0.35 | 0.758 | — | — | |
| TMVMPre-training dataset=W2M+C3M, Evaluation Protocol=Fine-tune2022.07 | 0.642 | 0.362 | 0.757 | 3 | — | |
| LAVENDER# PT Data=5M, Inference Setting=Fine-tuning2022.12 | 0.638 | 0.378 | 0.75 | — | — | |
| FlorenceZero-shot=true, Pre-training Data=900M2023.02 | 0.638 | 0.376 | 0.726 | — | — | |
| FlorenceSample=900M, Zero-shot=true, Modality=Vision-only2023.05 | 0.638 | 0.376 | 0.726 | — | — | |
| FlorenceDSize=900M, Params=637M, Zero-shot=true2024.01 | 0.638 | — | 0.726 | — | — | |
| MILESPre-training dataset=W2M+C3M, Evaluation Protocol=Fine-tune2022.07 | 0.636 | 0.377 | 0.738 | 3 | — | |
| MILESExample=5M2023.06 | 0.636 | 0.377 | 0.738 | — | — | |
| MILES#Example=5.5M, Mod=V2023.04 | 0.636 | 0.377 | 0.738 | — | — | |
| OA-TransPre-training dataset=W2M+C3M, Evaluation Protocol=Fine-tune2022.07 | 0.634 | 0.358 | 0.765 | 3 | — | |
| OA-TransExample=5M2023.06 | 0.634 | 0.358 | 0.765 | — | — | |
| UMT-LSample=425M, Zero-shot=true, Modality=Vision-only2023.05 | 0.634 | 0.407 | 0.718 | — | — | |
| OA-Trans#Example=5.5M, Mod=V2023.04 | 0.634 | 0.358 | 0.765 | — | — | |
| VIOLETPre-training dataset=W2M+C3M+Y180M, Evaluation Protocol=Fine-tune2022.07 | 0.63 | 0.345 | 0.734 | — | — | |
| VIOLET# PT Data=183M, Inference Setting=Fine-tuning2022.12 | 0.63 | 0.345 | 0.734 | — | — | |
| OmniVLZero-shot=true, Pre-training Data=18M2023.02 | 0.63 | 0.42 | 0.73 | — | — | |
| OmniVLSample=18M, Zero-shot=true, Modality=Vision-only2023.05 | 0.63 | 0.42 | 0.73 | — | — | |
| VIOLET#Example=186M, Mod=V2023.04 | 0.63 | 0.345 | 0.734 | — | — | |
| TVTSv2DSize=8.5M, Params=1B, Zero-shot=true2024.01 | 0.624 | — | 0.732 | 3 | — | |
| ImageBindDSize=3M, Params=1B, Zero-shot=true2024.01 | 0.618 | — | 0.7 | — | — | |
| Frozen#Example=6.1M, Mod=V2023.04 | 0.615 | 0.325 | 0.712 | — | — | |
| AlignPrompt#PT=5M, #Train Frame=82022.06 | 0.607 | 0.339 | 0.732 | — | — | |
| ALPROPre-training dataset=W2M+C3M, Evaluation Protocol=Fine-tune2022.07 | 0.607 | 0.339 | 0.732 | 3 | — | |
| ALPRO# PT Data=5M, Inference Setting=Fine-tuning2022.12 | 0.607 | 0.339 | 0.732 | — | — | |
| HiTeAZero-shot=true, Pre-training Data=17M2023.02 | 0.6 | 0.344 | 0.699 | — | — | |
| HiTeASample=17M, Zero-shot=true, Modality=Vision-only2023.05 | 0.6 | 0.344 | 0.699 | — | — | |
| SRL-CLIPBackbone=ViT-B/16, DSize=23k, Params=150M, Zero-shot=true2024.01 | 0.597 | — | 0.718 | 3 | — | |
| Frozen#PT=5M, #Train Frame=42022.06 | 0.595 | 0.31 | 0.705 | — | — | |
| FrozenPre-training dataset=W2M+C3M, Evaluation Protocol=Fine-tune2022.07 | 0.595 | 0.31 | 0.705 | 3 | — |