Audio-Visual Event Localization on AVE
85AccuracyTB-AVA (Ours)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| TB-AVA (Ours)Visual Encoder=SigLIP2-L/16, Audio Encoder=BEATs, Trainable Params (%)=16.7, Total Params (M)=488.12026.05 | 85 | — | |
| Crab+2026.03 | 83.58 | — | |
| MoLT2026.03 | 83.5 | — | |
| MoLTVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Trainable Params (%)=6.2, Total Params (M)=290.32026.05 | 83.5 | — | |
| MettleVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Trainable Params (%)=23.0, Total Params (M)=338.02026.05 | 83.3 | — | |
| TB-AVA without GSMVisual Encoder=SigLIP2-L/16, Audio Encoder=BEATs, Trainable Params (%)=16.4, Total Params (M)=486.52026.05 | 83.1 | — | |
| DG-SCTVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Trainable Params (%)=43.6, Total Params (M)=461.32026.05 | 81.2 | — | |
| LAVISHVisual Encoder=Swin-V2-L (frozen), Audio Encoder=(shared) (frozen), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=None, Trainable Params (M)=10.1, Total Params (M)=238.82022.12 | 81.1 | — | |
| LAVisHVisual Encoder=Swin-V2-L (shared), Audio Encoder=Swin-V2-L (shared), Trainable Params (%)=4.1, Total Params (M)=238.82026.05 | 80.8 | — | |
| UWAVBackbone=CLIP, R(2+1)D-18, and CLAP2025.05 | 80.6 | — | |
| CMBSImplementation=Improved implementation, Visual Encoder=Swin-V2-L (frozen), Audio Encoder=VGGish (fully fine-tuned), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=14.1, Total Params (M)=315.22022.12 | 80.4 | — | |
| VALORFeatures=CLAP, CLIP, R(2+1)D2023.05 | 80.4 | — | |
| VALORBackbone=CLIP, R(2+1)D-18, and CLAP2025.05 | 80.4 | — | |
| Crab2025.03 | 80.15 | — | |
| AVMoEVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Trainable Params (%)=34.9, Total Params (M)=404.02026.05 | 80 | — | |
| DPNetVisual Encoder=VGG-19, Audio Encoder=VGGish, Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet2022.12 | 79.7 | — | |
| CMBSVisual Encoder=ResNet-152 (frozen), Audio Encoder=VGGish (fully fine-tuned), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=14.4, Total Params (M)=216.72022.12 | 79.7 | — | |
| CMBSVisual Encoder=ResNet-151, Audio Encoder=VGG-like, Trainable Params (%)=6.6, Total Params (M)=216.72026.05 | 79.7 | — | |
| CMBSImplementation=Improved implementation, Visual Encoder=Swin-V2-L (fully fine-tuned), Audio Encoder=VGGish (fully fine-tuned), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=243.1, Total Params (M)=315.22022.12 | 79.6 | — | |
| CoPLVisual Encoder=ViT-L/16 (shared), Audio Encoder=ViT-L/16 (shared), Trainable Params (%)=1.5, Total Params (M)=332.82026.05 | 79.2 | — | |
| LAVISHVisual Encoder=Swin-V2-B (frozen), Audio Encoder=(shared) (frozen), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=None, Trainable Params (M)=5, Total Params (M)=114.22022.12 | 78.8 | — | |
| CMRANVisual Encoder=ResNet-152 (frozen), Audio Encoder=VGGish (fully fine-tuned), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=15.9, Total Params (M)=148.22022.12 | 78.3 | — | |
| CMRANVisual Encoder=ResNet-151, Audio Encoder=VGG-like, Trainable Params (%)=10.7, Total Params (M)=148.22026.05 | 78.3 | — | |
| LAVISHVisual Encoder=ViT-L-16 (frozen), Audio Encoder=(shared) (frozen), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=None, Trainable Params (M)=14.5, Total Params (M)=340.12022.12 | 78.1 | — | |
| LAVisHVisual Encoder=ViT-L/16 (shared), Audio Encoder=ViT-L/16 (shared), Trainable Params (%)=4.3, Total Params (M)=340.12026.05 | 78.1 | — | |
| PSPVisual Encoder=VGG-19 (fully fine-tuned), Audio Encoder=VGGish (fully fine-tuned), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=1.7, Total Params (M)=217.42022.12 | 77.8 | — | |
| MM-PyramidVisual Encoder=ResNet-152 (frozen), Audio Encoder=VGGish (fully fine-tuned), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=44, Total Params (M)=176.32022.12 | 77.8 | — | |
| MBTVisual Encoder=ViT-B-16 (frozen), Audio Encoder=AST (frozen), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=172, Total Params (M)=1722022.12 | 77.8 | — | |
| MM-Pyramid2026.03 | 77.8 | — | |
| MBTVisual Encoder=ViT-B/16, Audio Encoder=AST, Trainable Params (%)=100, Total Params (M)=172.02026.05 | 77.8 | — | |
| AVTVisual Encoder=VGG-19 (fully fine-tuned), Audio Encoder=VGGish (fully fine-tuned), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=15.8, Total Params (M)=231.52022.12 | 76.8 | — | |
| CMBSFeatures=VGG-like, Res-1512023.05 | 76 | — | |
| AVT2026.03 | 75.8 | — | |
| AVSDNVisual Encoder=ResNet-152 (frozen), Audio Encoder=VGGish (fully fine-tuned), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=8, Total Params (M)=140.32022.12 | 75.4 | — | |
| LAVISHVisual Encoder=ViT-B-16 (frozen), Audio Encoder=(shared) (frozen), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=None, Trainable Params (M)=4.7, Total Params (M)=107.22022.12 | 75.3 | — | |
| CMRANFeatures=VGG-like, Res-1512023.05 | 75.3 | — | |
| HANFeatures=CLAP, CLIP, R(2+1)D2023.05 | 75.3 | — | |
| HANBackbone=CLIP, R(2+1)D-18, and CLAP2025.05 | 75.3 | — | |
| CMBSFeatures=VGG-like, VGG-192023.05 | 74.2 | — | |
| AVSDNFeatures=VGG-like, Res-1512023.05 | 74.2 | — | |
| AVELVisual Encoder=ResNet-152 (frozen), Audio Encoder=VGGish (fully fine-tuned), Visual Pretrain Dataset=ImageNet, Audio Pretrain Dataset=AudioSet, Trainable Params (M)=3.7, Total Params (M)=1362022.12 | 74 | — | |
| PSPFeatures=VGG-like, VGG-192023.05 | 73.5 | — | |
| CMRANFeatures=VGG-like, VGG-192023.05 | 72.9 | — | |
| AVELVisual Encoder=VGG-19, Audio Encoder=VGG-like, Trainable Params (%)=2.7, Total Params (M)=136.02026.05 | 72.7 | — | |
| AVELFeatures=VGG-like, Res-1512023.05 | 71.6 | — | |
| CMANFeatures=VGG-like, VGG-192023.05 | 70.4 | — | |
| AVTFeatures=VGG-like, VGG-192023.05 | 70.2 | — | |
| AVINFeatures=VGG-like, VGG-192023.05 | 69.4 | — | |
| AVRBFeatures=VGG-like, VGG-192023.05 | 68.9 | — | |
| AVSDNFeatures=VGG-like, VGG-192023.05 | 67.3 | — | |
| AVELFeatures=VGG-like, VGG-192023.05 | 66.7 | — | |
| DG-SCTProtocol=Zero-shot2025.03 | 64.7 | — | |
| Drop-DTWLearning Paradigm=Unsupervised2021.08 | — | 41.1 | |
| OTAMLearning Paradigm=Unsupervised2021.08 | — | 37.5 | |
| SmoothDTWLearning Paradigm=Unsupervised2021.08 | — | 39.8 | |
| SupervisedLearning Paradigm=Supervised2021.08 | — | 44.8 |