Video-to-Text Retrieval on MSR-VTT (1k-A)
84.1Recall@5SMs (ours)
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| SMs (ours)Category=Zero-shot, Audio=yes, CLIP encoder=ViT-B/322022.04 | 84.1 | 60.7 | 90.6 | 1 | — | |
| CLIP via [67]Category=Zero-shot, Audio=no, CLIP encoder=ViT-B/322022.04 | 82.5 | 58 | 90.2 | 1 | — | |
| HVP-NetBackbone=CLIP ViT-B/32, Frames=12, Resolution=224x2242026.01 | 80.5 | 50.5 | 91.2 | 2 | 4.4 | |
| DRL (ViT-B/16 + QB-n)Category=Finetuning, Audio=no, CLIP encoder=ViT-B/162022.04 | 79.9 | 56.2 | 87.4 | 1 | — | |
| RAPBackbone=CLIP-ViT-B/16, Trainable Params (MB)=1.06, DSL post-processing=true2024.05 | 78.7 | 51.6 | 86.9 | — | 8 | |
| MUSE2026.01 | 77.8 | 49.7 | 86.5 | 2 | 7.4 | |
| CLIP2TVCategory=Finetuning, Audio=no, CLIP encoder=ViT-B/162022.04 | 77.4 | 54.1 | 85.7 | 1 | — | |
| X-CLIP2026.01 | 76.8 | 48.9 | 84.5 | 2 | 8.1 | |
| RAPBackbone=CLIP-ViT-B/16, Trainable Params (MB)=1.062024.05 | 76.4 | 45.3 | 84.8 | — | 9.1 | |
| TS2-NetBackbone=ViT16, Patch Size=16x162022.07 | 75.9 | 46.6 | 84.9 | 2 | 8.9 | |
| CLIP2TVPatch Size=16x162022.07 | 75.4 | 46.5 | 84.9 | 2 | 10.2 | |
| CLIP2TV-ViT16Type=Ours, Backbone=ViT-16, Similarity Distillation=true2021.11 | 75 | 46.9 | 85.1 | 2 | 10 | |
| CAMoECategory=Finetuning, Audio=no, CLIP encoder=ViT-B/322022.04 | 74.3 | 49.1 | 84.3 | 2 | — | |
| TS2-Net2022.07 | 74.1 | 45.3 | 83.7 | 2 | 9.2 | |
| BiHSSP2026.01 | 74.1 | 48 | 83.5 | 2 | 9 | |
| MV-AdapterBackbone=CLIP-ViT-B/16, Trainable Params (MB)=3.872024.05 | 74 | 45.6 | 83.8 | — | — | |
| DRL (ViT-B/32)Category=Finetuning, Audio=no, CLIP encoder=ViT-B/322022.04 | 73.9 | 45.3 | 83.3 | 2 | — | |
| CLIP2TV+SDType=Ours, Backbone=ViT-32, Similarity Distillation=true2021.11 | 73 | 43.9 | 82.8 | 2 | 11.1 | |
| CAMOE2022.07 | 72.4 | 45.1 | 83.1 | 2 | 10 | |
| CLIP2VideoCategory=Finetuning, Audio=no, CLIP encoder=ViT-B/322022.04 | 72.3 | 43.5 | 82.1 | 2 | — | |
| CLIP2Video2022.07 | 72.3 | 43.5 | 82.1 | 2 | 10.2 | |
| CLIP2VideoType=CLIP-based2021.11 | 72.3 | 43.5 | 82.1 | 2 | 10.2 | |
| CLIP2VideoTest-set=1k-A2021.06 | 72.3 | 43.5 | 82.1 | 2 | 10.2 | |
| CLIP4ClipBackbone=CLIP-ViT-B/16, Trainable Params (MB)=149.622024.05 | 72.2 | 44.9 | 81.8 | — | 10.4 | |
| RAPType=Adapter, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=1.062024.05 | 71.9 | 44 | 82.4 | — | 10.1 | |
| CLIP4Clip2022.07 | 70.9 | 42.7 | 80.6 | 2 | 11.6 | |
| CLIP4Clip-seqTransfType=CLIP-based2021.11 | 70.9 | 42.7 | 80.6 | 2 | 11.6 | |
| CLIP2TVType=Ours, Backbone=ViT-32, Similarity Distillation=false2021.11 | 70.9 | 43.9 | 82.2 | 2 | 12 | |
| CLIP4Clip2026.01 | 70.9 | 42.7 | 80.6 | 2 | 11.6 | |
| CLIP4Clip-seqTransfTest-set=1k-A, aggregation=temporal transformer2021.06 | 70.9 | 42.7 | 80.6 | 2 | 11.6 | |
| CLIP4ClipCategory=Finetuning, Audio=no, CLIP encoder=ViT-B/322022.04 | 70.5 | 43.1 | 81.2 | 2 | — | |
| CLIP4Clip-meanPType=CLIP-based2021.11 | 70.5 | 43.1 | 81.2 | 2 | 12.4 | |
| CLIP4Clip-meanPTest-set=1k-A, aggregation=mean pooling2021.06 | 70.5 | 43.1 | 81.2 | 2 | 12.4 | |
| VoP-CType=Prompt, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=14.302024.05 | 70.1 | 42.3 | 81.1 | — | 11.4 | |
| CLIP4ClipType=Fine-tune, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=151.282024.05 | 70 | 43.9 | 81.4 | — | 11.7 | |
| VPTType=Prompt, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=63.43, Trainable text-encoder=true2024.05 | 70 | 40.9 | 79.2 | — | 12.5 | |
| VoP-PType=Prompt, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=0.502024.05 | 70 | 42.5 | 79.9 | — | 12.4 | |
| ST-AdapterType=Adapter, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=77.45, Trainable text-encoder=true2024.05 | 70 | 42.1 | 81.2 | — | 11.4 | |
| LoRAType=Adapter, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=0.492024.05 | 70 | 42.1 | 79.8 | — | 13.5 | |
| SSFType=Adapter, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=0.342024.05 | 68.6 | 42 | 80.2 | — | 13.2 | |
| CoOpType=Prompt, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=0.022024.05 | 66.6 | 41 | 77.4 | — | 13.4 | |
| ST-AdapterType=Adapter, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=14.222024.05 | 64.5 | 37.1 | 75.9 | — | 19.7 | |
| VPTType=Prompt, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=0.212024.05 | 62.8 | 36.5 | 74.3 | — | 20 | |
| HIT-pretrained2022.07 | 62.7 | 32.1 | 74.1 | 3 | — | |
| HIT-pretrainedType=Others2021.11 | 62.7 | 32.1 | 74.1 | 3 | — | |
| HIT-pretrainedTest-set=1k-A2021.06 | 62.7 | 32.1 | 74.1 | 3 | — | |
| T2VLAD2022.07 | 60 | 31.8 | 71.1 | 3 | — | |
| SSBTraining=HT100M + MSR-VTT2021.02 | 58.6 | 28.5 | 71.6 | 3 | — | |
| SSBCategory=Finetuning, Audio=no2022.04 | 58.6 | 28.5 | 71.6 | 3 | — | |
| MMTTraining=HT100M + MSR-VTT2021.02 | 57.5 | 27 | 69.7 | 3.7 | — | |
| MMT2022.07 | 57.5 | 27 | 69.7 | 3.7 | 21.3 | |
| MMT-pretrainedType=Others2021.11 | 57.5 | 27 | 69.7 | 3.7 | 21.3 | |
| Multi-modal Transformerpre-trained=true2020.07 | 57.5 | — | — | 3.7 | 21.3 | |
| MMT-PretrainedTest-set=1k-A2021.06 | 57.5 | 27 | 69.7 | 3.7 | 21.3 | |
| Multi-modal Transformerpre-trained=false2020.07 | 56 | — | — | 4 | 23.6 | |
| SUPPORT-SET2022.07 | 55.1 | 26.6 | 67.5 | 3 | — | |
| SUPPORT-SETType=Others2021.11 | 55.1 | 26.6 | 67.5 | 3 | — | |
| SUPPORT-SETTest-set=1k-A2021.06 | 55.1 | 26.6 | 67.5 | 3 | — | |
| AVLnetTraining=HT100M + MSR-VTT2021.02 | 54.6 | 28.5 | 65.2 | 4 | — | |
| CLIPTraining=WIT2021.02 | 51.7 | 27.2 | 62.6 | 5 | — | |
| CLIP-straightType=CLIP-based2021.11 | 51.7 | 27.2 | 62.6 | 5 | — | |
| CLIPTest-set=1k-A2021.06 | 51.7 | 27.2 | 62.6 | 5 | — | |
| CETest-set=1k-A2019.07 | 50.3 | 20.6 | 64 | 5.3 | 25.1 | |
| CETraining=MSR-VTT2021.02 | 50.3 | 20.6 | 64 | 5.3 | — | |
| Collaborative ExpertsCategory=Finetuning, Audio=yes2022.04 | 50.3 | 20.6 | 64 | 5.3 | — | |
| CE2022.07 | 50.3 | 20.6 | 64 | 5.3 | 25.1 | |
| CEType=Others2021.11 | 50.3 | 20.6 | 64 | 5.3 | 25.1 | |
| CE2020.07 | 50.3 | — | — | 5.3 | 25.1 | |
| CETest-set=1k-A2021.06 | 50.3 | 20.6 | 64 | 5.3 | 25.1 | |
| CLIP4ClipType=Fine-tune, Backbone=CLIP-ViT-B/32, Trainable Params (MB)=0, Frozen visual encoder=true2024.05 | 50.1 | 26.5 | 61.7 | — | 39.9 | |
| HT100MTraining=HT100M + MSR-VTT2021.02 | 41.7 | 16.8 | 55.1 | 8 | — | |
| SSB (zero-shot)Category=Zero-shot, Audio=no2022.04 | 23 | 8.7 | 31.1 | 31 | — | |
| CLIP2022.07 | 22.7 | 11.3 | 29.2 | 5 | — | |
| Random baseline2020.07 | 0.5 | — | — | 500 | 500 |