Audio-visual event localization on AVE (test)
83.5AccuracyMoLT
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| MoLTVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=6.2, Total Params(M)=290.3, Memory (GB)=0.72025.11 | 83.5 | — | — | — | |
| MettleVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=23.0, Total Params(M)=338.0, Memory (GB)=1.52025.11 | 83.3 | — | — | — | |
| AVMoEVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=34.9, Total Params(M)=404.0, Memory (GB)=12.12025.11 | 82.6 | — | — | — | |
| AVMoEVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Trainable Params(%)=34.9, Total Params(M)=374.42026.02 | 82.6 | — | — | — | |
| CAE-AVVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Trainable Params(%)=43.5, Total Params(M)=460.92026.02 | 82.6 | — | — | — | |
| MettleVisual Encoder=Swin-V2-L (shared), Audio Encoder=Swin-V2-L (shared), Visual PT Dataset=ImageNet, Audio PT Dataset=None, Trainable Params(%)=37.2, Total Params(M)=364.4, Memory (GB)=1.82025.11 | 82.3 | — | — | — | |
| DG-SCTVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=43.6, Total Params(M)=461.3, Memory (GB)=13.12025.11 | 82.2 | — | — | — | |
| CAE-AVVisual Encoder=Swin-V2-L (shared), Audio Encoder=Swin-V2-L (shared), Trainable Params(%)=35.7, Total Params(M)=355.92026.02 | 82.2 | — | — | — | |
| DG-SCTVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Trainable Params(%)=43.6, Total Params(M)=461.32026.02 | 82.2 | — | — | — | |
| AVMoEVisual Encoder=Swin-V2-L (shared), Audio Encoder=Swin-V2-L (shared), Visual PT Dataset=ImageNet, Audio PT Dataset=None, Trainable Params(%)=39.4, Total Params(M)=374.4, Memory (GB)=16.82025.11 | 81.5 | — | — | — | |
| Mettle*Visual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=30.6, Total Params(M)=374.9, Memory (GB)=1.32025.11 | 81.5 | — | — | — | |
| AVMoEVisual Encoder=Swin-V2-L (shared), Audio Encoder=Swin-V2-L (shared), Trainable Params(%)=32.6, Total Params(M)=483.12026.02 | 81.5 | — | — | — | |
| LAVisHVisual Encoder=Swin-V2-L (shared), Audio Encoder=Swin-V2-L (shared), Visual PT Dataset=ImageNet, Audio PT Dataset=None, Trainable Params(%)=2.7, Total Params(M)=238.8, Memory (GB)=15.42025.11 | 81.1 | — | — | — | |
| LAVisHVisual Encoder=Swin-V2-L (shared), Audio Encoder=Swin-V2-L (shared), Trainable Params(%)=2.7, Total Params(M)=238.82026.02 | 81.1 | — | — | — | |
| Crab2025.03 | 80.15 | — | — | — | |
| CMBSVisual Encoder=ResNet-152, Audio Encoder=VGGish, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=6.6, Total Params(M)=216.72025.11 | 79.7 | — | — | — | |
| MoLTVisual Encoder=Swin-V2-L (shared), Audio Encoder=Swin-V2-L (shared), Visual PT Dataset=ImageNet, Audio PT Dataset=None, Trainable Params(%)=3.0, Total Params(M)=248.5, Memory (GB)=0.62025.11 | 79.7 | — | — | — | |
| AVMoEVisual Encoder=Swin-V2-B (shared), Audio Encoder=Swin-V2-B (shared), Trainable Params(%)=31.8, Total Params(M)=150.42026.02 | 79.4 | — | — | — | |
| CMBSVisual Encoder=ResNet-152, Audio Encoder=VGGish, Trainable Params(%)=6.6, Total Params(M)=216.72026.02 | 79.3 | — | — | — | |
| CAE-AVVisual Encoder=Swin-V2-B (shared), Audio Encoder=Swin-V2-B (shared), Trainable Params(%)=45.5, Total Params(M)=200.72026.02 | 79.2 | — | — | — | |
| LAVisHVisual Encoder=Swin-V2-B (shared), Audio Encoder=Swin-V2-B (shared), Trainable Params(%)=4.4, Total Params(M)=114.22026.02 | 78.8 | — | — | — | |
| AV-UnifiedSupervision protocol=Fully-Supervised2026.03 | 78.7 | — | — | — | |
| LAVisHVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=30.6, Total Params(M)=374.9, Memory (GB)=11.92025.11 | 78.6 | — | — | — | |
| LAVisHVisual Encoder=Swin-V2-L, Audio Encoder=HTS-AT, Trainable Params(%)=30.6, Total Params(M)=374.92026.02 | 78.6 | — | — | — | |
| CMRANVisual Encoder=ResNet-152, Audio Encoder=VGGish, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=10.7, Total Params(M)=148.22025.11 | 78.3 | — | — | — | |
| Mettle*Visual Encoder=Swin-V2-L (shared), Audio Encoder=Swin-V2-L (shared), Visual PT Dataset=ImageNet, Audio PT Dataset=None, Trainable Params(%)=37.0, Total Params(M)=363.3, Memory (GB)=1.32025.11 | 78 | — | — | — | |
| PSP2025.03 | 77.8 | — | — | — | |
| MM-Pyramid2025.03 | 77.8 | — | — | — | |
| MM-PyramidVisual Encoder=ResNet-152, Audio Encoder=VGGish, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=25.0, Total Params(M)=176.32025.11 | 77.8 | — | — | — | |
| PSPVisual Encoder=VGG-19, Audio Encoder=VGG-like, Trainable Params(%)=0.8, Total Params(M)=217.42026.02 | 77.8 | — | — | — | |
| PSPSupervision protocol=Fully-Supervised2026.03 | 77.8 | — | — | — | |
| MM-PyramidSupervision protocol=Fully-Supervised2026.03 | 77.8 | — | — | — | |
| MPN2025.03 | 77.6 | — | — | — | |
| CMRANSupervision protocol=Fully-Supervised2026.03 | 77.4 | — | — | — | |
| AVT2025.03 | 75.8 | — | — | — | |
| AVTSupervision protocol=Fully-Supervised2026.03 | 75.8 | — | — | — | |
| AVSDNVisual Encoder=ResNet-152, Audio Encoder=VGGish, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=5.7, Total Params(M)=140.32025.11 | 75.4 | — | — | — | |
| AVINSupervision protocol=Fully-Supervised2026.03 | 75.2 | — | — | — | |
| AV-UnifiedSupervision protocol=Weakly-Supervised2026.03 | 74.2 | — | — | — | |
| AVELVisual Encoder=ResNet-152, Audio Encoder=VGGish, Visual PT Dataset=ImageNet, Audio PT Dataset=AudioSet, Trainable Params(%)=2.7, Total Params(M)=136.02025.11 | 74 | — | — | — | |
| AVELVisual Encoder=ResNet-152, Audio Encoder=VGGish, Trainable Params(%)=2.7, Total Params(M)=136.02026.02 | 74 | — | — | — | |
| MIMOt(s)=0.029±0.0022025.11 | 73.69 | 72.69 | 71.85 | — | |
| PSPSupervision protocol=Weakly-Supervised2026.03 | 73.5 | — | — | — | |
| MM-PyramidSupervision protocol=Weakly-Supervised2026.03 | 73.2 | — | — | — | |
| CMRANSupervision protocol=Weakly-Supervised2026.03 | 73 | — | — | — | |
| AVELSupervision protocol=Fully-Supervised2026.03 | 72.7 | — | — | — | |
| AGMt(s)=0.161±0.0022025.11 | 72.54 | 54.73 | 50.92 | — | |
| AVTSupervision protocol=Weakly-Supervised2026.03 | 70.2 | — | — | — | |
| AVINSupervision protocol=Weakly-Supervised2026.03 | 69.4 | — | — | — | |
| MSLt(s)=0.027±0.0022025.11 | 67.41 | 33.46 | 56.61 | — | |
| AVELSupervision protocol=Weakly-Supervised2026.03 | 66.7 | — | — | — | |
| CMLCLManners=unsupervised2024.11 | 63.2 | — | — | — | |
| fine-tuning baselineManners=zero-shot2024.11 | 61.9 | — | — | — | |
| training-free baselineManners=zero-shot2024.11 | 54.8 | — | — | — | |
| Audiot(s)=0.010±0.0012025.11 | — | 66.03 | — | — | |
| CoLABackbone (Vision/Audio)=ViT-B-16/SSLAM, Param Update=18.6, Total Parameters=211.6, Update Ratio=8.82026.04 | — | — | — | 79.1 | |
| CoLABackbone (Vision/Audio)=DINOv2-B-14/SSLAM, Param Update=18.6, Total Parameters=195.6, Update Ratio=9.5%2026.04 | — | — | — | 80.7 | |
| CoLABackbone (Vision/Audio)=DINOv2-L-14/SSLAM, Param Update=26.4, Total Parameters=421.2, Update Ratio=6.3%2026.04 | — | — | — | 81.1 | |
| LAVisHBackbone (Vision/Audio)=ViT-B-16 (Shared), Param Update=4.7, Total Parameters=107.2, Update Ratio=4.4%2026.04 | — | — | — | 75.3 | |
| LAVisHBackbone (Vision/Audio)=ViT-L-14 (Shared), Param Update=14.5, Total Parameters=340.1, Update Ratio=4.3%2026.04 | — | — | — | 78.1 | |
| STG-CMABackbone (Vision/Audio)=CLIP-B-16 (Shared), Param Update=11.5, Total Parameters=97.5, Update Ratio=11.8%2026.04 | — | — | — | 78.7 | |
| STG-CMABackbone (Vision/Audio)=CLIP-L-14 (Shared), Param Update=20.1, Total Parameters=323.6, Update Ratio=6.2%2026.04 | — | — | — | 83.3 | |
| Visualt(s)=0.011±0.0012025.11 | — | — | 63.82 | — |