Image-Text Retrieval on COCO (val)
60.1R@1DINOv2-ARL
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| DINOv2-ARLEvaluation Protocol=zero-shot2024.09 | 60.1 | — | — | |
| LAION-CLIPBackbone=ViT-L, Evaluation Protocol=zero-shot2024.09 | 59.7 | — | — | |
| DINOv2-MpNetEvaluation Protocol=zero-shot2024.09 | 58 | — | — | |
| CLIPvisual encoder=ViT-L/14-336, visual FLOPS=191.0G2023.12 | 57.9 | — | — | |
| SF-LCLIPvisual encoder=SF-LCLIP, tokens=144, visual FLOPS=33.9G2023.12 | 57 | — | — | |
| TinyCLIPvisual encoder=ViT-63M/32-224, visual FLOPS=2.0G2023.12 | 56.9 | — | — | |
| CLIPvisual encoder=ViT-L/14-224, visual FLOPS=81.1G2023.12 | 56.3 | — | — | |
| OpenAI-CLIPBackbone=ViT-L, Evaluation Protocol=zero-shot2024.09 | 56.3 | — | — | |
| SF-LCLIPvisual encoder=SF-LCLIP, tokens=128, visual FLOPS=30.0G2023.12 | 54.7 | — | — | |
| 3TPretraining=IN-21k, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 54.4 | — | — | |
| 3TPretraining=JFT, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 54.4 | — | — | |
| LiTPretraining=JFT, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 52.7 | — | — | |
| CLIPvisual encoder=ViT-B/16-224, visual FLOPS=17.5G2023.12 | 52.4 | — | — | |
| SF-LCLIPvisual encoder=SF-LCLIP, tokens=64, visual FLOPS=14.8G2023.12 | 52.3 | — | — | |
| Baseline (CLIP/ALIGN)Pretraining=IN-21k, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 51 | — | — | |
| CLIPvisual encoder=ViT-B/32-224, visual FLOPS=4.4G2023.12 | 50.1 | — | — | |
| LiTBackbone=L16L, Evaluation Protocol=zero-shot2024.09 | 48.5 | — | — | |
| SF-BCLIPvisual encoder=SF-BCLIP, tokens=49, visual FLOPS=3.8G2023.12 | 47.7 | — | — | |
| LiTPretraining=IN-21k, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 46.1 | — | — | |
| DINOv2-ARLEvaluation Protocol=zero-shot2024.09 | 45.1 | — | — | |
| LAION-CLIPBackbone=ViT-L, Evaluation Protocol=zero-shot2024.09 | 43 | — | — | |
| DINOv2-MpNetEvaluation Protocol=zero-shot2024.09 | 42.6 | — | — | |
| TinyCLIPvisual encoder=ViT-63M/32-224, visual FLOPS=2.0G2023.12 | 38.5 | — | — | |
| SF-LCLIPvisual encoder=SF-LCLIP, tokens=144, visual FLOPS=33.9G2023.12 | 38.3 | — | — | |
| 3TPretraining=JFT, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 37.9 | — | — | |
| 3TPretraining=IN-21k, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 37.7 | — | — | |
| CLIPvisual encoder=ViT-L/14-336, visual FLOPS=191.0G2023.12 | 37.1 | — | — | |
| LiTPretraining=JFT, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 36.7 | — | — | |
| CLIPvisual encoder=ViT-L/14-224, visual FLOPS=81.1G2023.12 | 36.5 | — | — | |
| OpenAI-CLIPBackbone=ViT-L, Evaluation Protocol=zero-shot2024.09 | 36.5 | — | — | |
| SF-LCLIPvisual encoder=SF-LCLIP, tokens=128, visual FLOPS=30.0G2023.12 | 36.4 | — | — | |
| SF-LCLIPvisual encoder=SF-LCLIP, tokens=64, visual FLOPS=14.8G2023.12 | 34.8 | — | — | |
| Baseline (CLIP/ALIGN)Pretraining=IN-21k, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 34.2 | — | — | |
| CLIPvisual encoder=ViT-B/16-224, visual FLOPS=17.5G2023.12 | 33 | — | — | |
| CLIPzero-shot=true2023.10 | 32.58 | 44.57 | 57.62 | |
| Ex-MCRzero-shot=true2023.10 | 32.58 | 44.57 | 57.62 | |
| SF-BCLIPvisual encoder=SF-BCLIP, tokens=49, visual FLOPS=3.8G2023.12 | 31.5 | — | — | |
| LiTBackbone=L16L, Evaluation Protocol=zero-shot2024.09 | 31.2 | — | — | |
| CLIPvisual encoder=ViT-B/32-224, visual FLOPS=4.4G2023.12 | 30.4 | — | — | |
| LiTPretraining=IN-21k, Model Scale=L, Training Dataset=Unfiltered WebLI, Evaluation Protocol=Zero-shot2023.05 | 27.8 | — | — | |
| ULIPzero-shot=true2023.10 | 22.92 | 34.42 | 46.33 | |
| ULIP v2zero-shot=true2023.10 | 22.92 | 34.42 | 46.33 | |
| C-MCRzero-shot=true2023.10 | 14.34 | 24.23 | 33.19 | |
| AudioCLIPZero-shot=true2023.10 | — | 20.14 | 27.42 | |
| C-MCRZero-shot=true2023.10 | — | 24.56 | 33.83 | |
| CLIPZero-shot=true2023.10 | — | 44.57 | 57.62 | |
| Ex-MCRZero-shot=true2023.10 | — | 44.57 | 57.62 | |
| WAV2CLIPZero-shot=true2023.10 | — | 44.57 | 57.62 |