Audio Classification on ESC50
96.58Top-1 AccPALM + ASPL*
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PALM + ASPL*Shots=16, Audio Encoder=CLAP-HTSAT, Text Encoder=CLIP (PENGI)2026.06 | 96.58 | — | |
| PALM + ASPLShots=16, Audio Encoder=CLAP-HTSAT, Text Encoder=CLIP (PENGI)2026.06 | 96.52 | — | |
| XKDPretrain=AS, Modality=Video + Audio, Evaluation Protocol=Fine-tuning2022.11 | 96.5 | — | |
| PALMShots=16, Audio Encoder=CLAP-HTSAT, Text Encoder=CLIP (PENGI)2026.06 | 96.33 | — | |
| PEAV LA-Enc Params.=1.1B, Data (M)=124, Sampling=30 FPS2025.12 | 96 | — | |
| PEAV BA-Enc Params.=.2B, Data (M)=124, Sampling=16 Frames2025.12 | 95.6 | — | |
| PaSST(SL)Pretrain=AS+IN, Modality=Audio + Image, Evaluation Protocol=Fine-tuning2022.11 | 95.5 | — | |
| CoCoOp + ASPLShots=16, Audio Encoder=CLAP-HTSAT, Text Encoder=CLIP (PENGI)2026.06 | 95.37 | — | |
| CoCoOp + ASPL*Shots=16, Audio Encoder=CLAP-HTSAT, Text Encoder=CLIP (PENGI)2026.06 | 95.32 | — | |
| PEAV SA-Enc Params.=.09B, Data (M)=124, Sampling=30 FPS2025.12 | 95.2 | — | |
| MS-CLAP23’A-Enc Params.=.08B, Data (M)=0.12025.12 | 95.1 | — | |
| PEAV SA-Enc Params.=.09B, Data (M)=124, Sampling=16 Frames2025.12 | 95.1 | — | |
| PEAV BA-Enc Params.=0.2B, Data (M)=124, Sampling=30 FPS2025.12 | 95.1 | — | |
| PEAV LA-Enc Params.=1.1B, Data (M)=124, Sampling=16 Frames2025.12 | 95 | — | |
| CoCoOpShots=16, Audio Encoder=CLAP-HTSAT, Text Encoder=CLIP (PENGI)2026.06 | 94.72 | — | |
| PEAV L-OODA-Enc Params.=1.1B, Data (M)=114, Sampling=30 FPS, OOD=true2025.12 | 94.7 | — | |
| PEAV L (PT)A-Enc Params.=1.1B, Data (M)=92, Sampling=30 FPS, PT=true2025.12 | 94.4 | — | |
| CoOp + ASPL*Shots=16, Audio Encoder=CLAP-HTSAT, Text Encoder=CLIP (PENGI)2026.06 | 94.35 | — | |
| CoOp + ASPLShots=16, Audio Encoder=CLAP-HTSAT, Text Encoder=CLIP (PENGI)2026.06 | 94.1 | — | |
| CoOpShots=16, Audio Encoder=CLAP-HTSAT, Text Encoder=CLIP (PENGI)2026.06 | 93.8 | — | |
| XKDPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Fine-tuning2022.11 | 93.6 | — | |
| XKDPretrain=AS, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 93.6 | — | |
| AudioMAEPretrain=AS, Modality=Audio, Evaluation Protocol=Fine-tuning2022.11 | 93.6 | — | |
| AFlamingo2A-Enc Params.=0.3B, Data (M)=82025.12 | 93.6 | — | |
| CLAPA-Enc Params.=.03B, Data (M)=32025.12 | 93.3 | — | |
| XKD-MATSPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Fine-tuning2022.11 | 92.9 | — | |
| XKD-MASPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Fine-tuning2022.11 | 92.7 | — | |
| CLAP-FusionA-Enc Params.=.03B, Data (M)=32025.12 | 92.2 | — | |
| CrissCrossPretrain=AS, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 90.5 | — | |
| MAE-ASTPretrain=AS, Modality=Audio, Evaluation Protocol=Fine-tuning2022.11 | 90 | — | |
| MaskSpecPretrain=AS, Modality=Audio, Evaluation Protocol=Fine-tuning2022.11 | 89.6 | — | |
| XKDPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 89.4 | — | |
| AVIDPretrain=AS, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 89.2 | — | |
| SS-ASTPretrain=AS, Modality=Audio, Evaluation Protocol=Fine-tuning2022.11 | 88.8 | — | |
| XKD-MATSPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 88.7 | — | |
| GDTPretrain=AS, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 88.5 | — | |
| LangBindA-Enc Params.=0.3B, Data (M)=102025.12 | 88.3 | — | |
| XKD-MASPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 87.3 | — | |
| CrissCrossPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 86.8 | — | |
| VATT + GRPre-training Dataset=HT100M + AudioSet + YT8M, Cross-Modality Gradient Realignment (GR)=true2022.11 | 85 | 98 | |
| VATTPretrain=AS+HT, Modality=Video + Audio + Text, Evaluation Protocol=Linear2022.11 | 84.7 | — | |
| VATT + BothPre-training Dataset=HT100M + AudioSet + YT8M, Cross-Modality Gradient Realignment (GR)=true, Gradient-based Curriculum Learning (CL)=true2022.11 | 84.5 | 98 | |
| Ours w/ PointBindSetting=Fine-tuning2024.03 | 84.01 | — | |
| VATT + RW (VA)Pre-training Dataset=HT100M + AudioSet + YT8M, Gradient Re-weighting (RW)=Video-Audio2022.11 | 83.5 | 97.25 | |
| VATT + CLPre-training Dataset=HT100M + AudioSet + YT8M, Gradient-based Curriculum Learning (CL)=true2022.11 | 83.5 | 97.75 | |
| ImageBindSetting=Fine-tuning, Protocol=linear2024.03 | 83.4 | — | |
| PointBindSetting=Fine-tuning, Protocol=linear2024.03 | 83.4 | — | |
| PointBind (+Event)Setting=Fine-tuning2024.03 | 83.4 | — | |
| VATT + RW (VT)Pre-training Dataset=HT100M + AudioSet + YT8M, Gradient Re-weighting (RW)=Video-Text2022.11 | 82.25 | 96.75 | |
| VATT + CLPre-training Dataset=HT100M + AudioSet, Gradient-based Curriculum Learning (CL)=true2022.11 | 82 | 97.25 | |
| VATT + BothPre-training Dataset=HT100M + AudioSet, Cross-Modality Gradient Realignment (GR)=true, Gradient-based Curriculum Learning (CL)=true2022.11 | 82 | 97 | |
| VATT + RW (VA)Pre-training Dataset=HT100M + AudioSet, Gradient Re-weighting (RW)=Video-Audio2022.11 | 81.75 | 97.25 | |
| VATT + GRPre-training Dataset=HT100M + AudioSet, Cross-Modality Gradient Realignment (GR)=true2022.11 | 81.5 | 97 | |
| VATTPre-training Dataset=HT100M + AudioSet2022.11 | 81.2 | — | |
| VATT-MAPretrain=AS+HT, Modality=Video + Audio + Text, Evaluation Protocol=Linear2022.11 | 81.2 | — | |
| STICAPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 81.1 | — | |
| VATT + RW (VT)Pre-training Dataset=HT100M + AudioSet, Gradient Re-weighting (RW)=Video-Text2022.11 | 80.75 | 96.75 | |
| VATTPre-training Dataset=HT100M + AudioSet + YT8M2022.11 | 80.75 | 97 | |
| AVTSPretrain=AS, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 80.6 | — | |
| AVIDPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 79.1 | — | |
| XDCPretrain=K400, Modality=Video + Audio, Evaluation Protocol=Linear2022.11 | 78 | — | |
| M2D-CLAPA-Enc Params.=.09B, Data (M)=22025.12 | 77.8 | — | |
| VATT + CLPre-training Dataset=HT100M, Gradient-based Curriculum Learning (CL)=true2022.11 | 72.5 | 94.25 | |
| VATT + BothPre-training Dataset=HT100M, Cross-Modality Gradient Realignment (GR)=true, Gradient-based Curriculum Learning (CL)=true2022.11 | 72.05 | 94.16 | |
| Ours w/ PointBindSetting=Zero-shot2024.03 | 71.7 | — | |
| VATT + RW (VA)Pre-training Dataset=HT100M, Gradient Re-weighting (RW)=Video-Audio2022.11 | 71.56 | 93.02 | |
| VATTPre-training Dataset=HT100M2022.11 | 71.5 | 91.75 | |
| VATT + RW (VT)Pre-training Dataset=HT100M, Gradient Re-weighting (RW)=Video-Text2022.11 | 71.29 | 91.38 | |
| VATT + GRPre-training Dataset=HT100M, Cross-Modality Gradient Realignment (GR)=true2022.11 | 69 | 93 | |
| ImageBindA-Enc Params.=.09B, Data (M)=32025.12 | 67.4 | — | |
| ImageBindSetting=Zero-shot2024.03 | 66.9 | — | |
| PointBindSetting=Zero-shot2024.03 | 66.9 | — | |
| PointBind (+Event)Setting=Zero-shot2024.03 | 66.9 | — |