Cross-modal retrieval on AudioCaps (test)
59.1R@1OmniBind
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| OmniBindmultimodal query=true2024.10 | 59.1 | 83.8 | — | — | — | — | |
| OmniBind2024.10 | 56 | 80.8 | — | — | — | — | |
| CART2024.06 | 49.8 | 79.8 | 86.8 | 62 | — | — | |
| Laion-CLAPmultimodal query=true2024.10 | 45.3 | 80.3 | — | — | — | — | |
| Laion-CLAP2024.10 | 44 | 79.5 | — | — | — | — | |
| AVRPretraining=VideoCC3M, Modality=Audio+Visual, Zero-shot=false2022.04 | 43.2 | — | 88.9 | — | — | — | |
| ONE-PEACETraining Status=Pretrained2024.06 | 42.5 | 77.5 | 88.4 | — | — | — | |
| HTSAT-BERT2024.06 | 42.2 | 76.5 | 87.1 | — | — | — | |
| AVRPretraining=None, Modality=Audio+Visual, Zero-shot=false2022.04 | 41.4 | — | 85.3 | — | — | — | |
| Multi-TTAAudio Encoder=PANNs (ResNet38), Text Encoder=BERT, Data Augmentation=PairMix, Test-Time Augmentation=Multi-TTA2022.10 | 40.2 | 74 | 87.2 | — | 97.6 | 6.3 | |
| VALORVariant=B2024.06 | 40.1 | 73.9 | 83.1 | — | — | — | |
| PairMixAudio Encoder=PANNs (ResNet38), Text Encoder=BERT, Data Augmentation=PairMix2022.10 | 40 | 73.1 | 86.8 | — | 97.3 | 6.6 | |
| BaselineAudio Encoder=PANNs (ResNet38), Text Encoder=BERT2022.10 | 37.9 | 71 | 82.7 | — | 97.4 | 8.8 | |
| CLAP-HTSAT2024.06 | 36.1 | 71.8 | 83.9 | — | — | — | |
| AVRPretraining=VideoCC3M, Modality=Audio, Zero-shot=false2022.04 | 35.5 | — | 84.5 | — | — | — | |
| AVRPretraining=HowTo100M, Modality=Audio, Zero-shot=false2022.04 | 33.7 | — | 83.2 | — | — | — | |
| AVRPretraining=None, Modality=Audio, Zero-shot=false2022.04 | 32 | — | 82.3 | — | — | — | |
| SOTAPretraining=None, Modality=Audio+Visual, Zero-shot=false2022.04 | 28.1 | — | 79 | — | — | — | |
| SOTAPretraining=None, Modality=Audio, Zero-shot=false2022.04 | 24.3 | — | 72.1 | — | — | — | |
| LanguageBindModality=Audio2024.06 | 15.2 | 49.8 | 67.6 | 29.8 | — | — | |
| AVRPretraining=VideoCC3M, Modality=Audio+Visual, Zero-shot=true2022.04 | 10.6 | — | 45.2 | — | — | — | |
| AVRPretraining=VideoCC3M, Modality=Audio, Zero-shot=true2022.04 | 8.7 | — | 37.7 | — | — | — | |
| AVRPretraining=HowTo100M, Modality=Audio, Zero-shot=true2022.04 | 1.4 | — | 6.5 | — | — | — |