Holistic Evaluation of Audio Representations (HEAR)
84.4HEAR AverageUSAD 2.0+ XLarge+
Evaluation results
| Method | Links | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| USAD 2.0+ XLarge+Params=695M2026.06 | 84.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| USAD 2.0+ XXLarge+Params=1036M2026.06 | 84.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| USAD 2.0+ Large+Params=336M2026.06 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPEARs+a XLarge# Params=600M, Pre-train Data=197k, Feature extraction protocol=Concatenation of all layers2025.10 | 83.41 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 84.85 | 79.75 | 85.43 | |
| USAD 2.0 LargeParams=336M2026.06 | 82.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPEAR XLargeParams=~600M2026.06 | 82.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| USAD 2.0 XLargeParams=695M2026.06 | 82.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPEARs+a Large# Params=327M, Pre-train Data=97k, Feature extraction protocol=Concatenation of all layers2025.10 | 82.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 84.33 | 78.88 | 84.21 | |
| Multi-expert Encoder (Self-supervised)Params=734M, Components=WavLM + ATST + MuQ2026.06 | 82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| USAD 2.0 BaseParams=97M2026.06 | 81.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPEAR LargeParams=~300M2026.06 | 81.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Multi-expert Encoder (Supervised)Params=1274M, Components=Whisper + AF32026.06 | 81.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Dasheng 1.2B# Params=1.2B, Pre-train Data=272k, Feature extraction protocol=Final layer2025.10 | 81.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 83.2 | 75.72 | 84.86 | |
| SPEARs+a Base# Params=94M, Pre-train Data=97k, Feature extraction protocol=Concatenation of all layers2025.10 | 81.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 82.55 | 77.45 | 83.61 | |
| Dasheng 0.6B# Params=600M, Pre-train Data=272k, Feature extraction protocol=Final layer2025.10 | 81.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 82.95 | 74.82 | 84.73 | |
| USAD 2.0 SmallParams=25M2026.06 | 81 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPEAR BaseParams=~90M2026.06 | 80.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPEARs+a XLarge# Params=600M, Pre-train Data=197k, Feature extraction protocol=Final layer2025.10 | 80.07 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 82.3 | 76.97 | 81.29 | |
| USAD Large# Params=330M, Pre-train Data=126k, Feature extraction protocol=Final layer2025.10 | 79.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 81.97 | 74.48 | 81.7 | |
| Dasheng-Base# Params=86M, Pre-train Data=272k, Feature extraction protocol=Final layer2025.10 | 79.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80.18 | 72.48 | 84 | |
| SPEARs+a Large# Params=327M, Pre-train Data=97k, Feature extraction protocol=Final layer2025.10 | 79.26 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 81.1 | 76.47 | 80.42 | |
| SPEARa Large# Params=327M, Pre-train Data=13k, Feature extraction protocol=Final layer2025.10 | 79.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 83.58 | 72.7 | 81.85 | |
| SPEARa Large# Params=327M, Pre-train Data=5k, Feature extraction protocol=Final layer2025.10 | 78.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 78.16 | 72.94 | 81.8 | |
| SPEARs+a Base# Params=94M, Pre-train Data=97k, Feature extraction protocol=Final layer2025.10 | 77.83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80.6 | 73.95 | 79.35 | |
| USAD Base# Params=94M, Pre-train Data=126k, Feature extraction protocol=Final layer2025.10 | 77.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80.67 | 73.72 | 79.31 | |
| SPEARa Base# Params=94M, Pre-train Data=13k, Feature extraction protocol=Final layer2025.10 | 77.01 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80.33 | 69.87 | 80.33 | |
| SPEARa Base# Params=94M, Pre-train Data=5k, Feature extraction protocol=Final layer2025.10 | 76.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 77.83 | 69.74 | 80.61 | |
| Dasheng 1.2B# Params=1.2B, Pre-train Data=5k, Feature extraction protocol=Final layer2025.10 | 74.87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SPEARs Large# Params=327M, Pre-train Data=84k, Feature extraction protocol=Final layer2025.10 | 72.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 72.74 | 74.8 | 71.68 | |
| SPEARs Base# Params=94M, Pre-train Data=84k, Feature extraction protocol=Final layer2025.10 | 72.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 73.09 | 73.41 | 70.66 | |
| Dasheng 0.6B# Params=600M, Pre-train Data=5k, Feature extraction protocol=Final layer2025.10 | 71.75 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BEATs# Params=90M, Pre-train Data=5k, Feature extraction protocol=Final layer2025.10 | 71.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 73.23 | 62.4 | 77.52 | |
| Dasheng-Base# Params=86M, Pre-train Data=5k, Feature extraction protocol=Final layer2025.10 | 70.43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WavLM Large# Params=317M, Pre-train Data=94k, Feature extraction protocol=Final layer2025.10 | 69.65 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 72.86 | 72.69 | 65.77 | |
| BYOL-S++Pre-training dataset=BYOL-S++2022.06 | 66.3 | 66.4 | 80 | 93.2 | 95 | 15.4 | 47.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid BYOL-S/CvT2022.06 | 66.3 | 67.2 | 83.5 | 92.6 | 95.8 | 16.3 | 42.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid BYOL-S/CvTalpha:beta ratio=1:12022.06 | 66.3 | 67.2 | 83.5 | 92.6 | 95.8 | 16.3 | 42.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-SWindow size (s)=22022.06 | 65.8 | 65.6 | 81.8 | 92.2 | 93.4 | 14.3 | 47.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-SWindow size (s)=0.952022.06 | 65.2 | 65.5 | 83 | 91.9 | 94.5 | 14.8 | 44.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid BYOL-S/Default2022.06 | 65.2 | 66.1 | 81.6 | 91.6 | 93.8 | 14.2 | 43.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid BYOL-S/CvTalpha:beta ratio=2:12022.06 | 65 | 65.8 | 81.2 | 92.7 | 94.7 | 15.8 | 39.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-SEncoder=CvT2022.06 | 64.9 | 66.4 | 84.8 | 92 | 93.1 | 16.2 | 37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid BYOL-S/CvTalpha:beta ratio=2:32022.06 | 64.9 | 66 | 84.1 | 91.9 | 93.8 | 14.6 | 38.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-SWindow size (s)=0.52022.06 | 64.8 | 66.5 | 86 | 91.2 | 93.4 | 14.2 | 43 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid BYOL-S/CvTalpha:beta ratio=4:12022.06 | 64.8 | 66.5 | 80.6 | 93.1 | 94.6 | 16.2 | 38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-SPre-training dataset=BYOL-S2022.06 | 64.7 | 66.4 | 78.5 | 92.6 | 94.3 | 15.1 | 41.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-SWindow size (s)=1.4252022.06 | 64.7 | 65.7 | 81 | 90 | 93.1 | 16 | 43.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-SEncoder=Default2022.06 | 64.7 | 66.4 | 78.6 | 92.6 | 94.3 | 15.1 | 41.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid BYOL-S/CvTalpha:beta ratio=3:22022.06 | 64.3 | 66.7 | 82.7 | 92.5 | 94.8 | 15.2 | 34 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid BYOL-S/CvTalpha:beta ratio=1:22022.06 | 63.2 | 64.4 | 83.5 | 89.6 | 92.7 | 13.6 | 35.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| WavLM Base+# Params=95M, Pre-train Data=94k, Feature extraction protocol=Final layer2025.10 | 62.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 57.28 | 68.14 | 61.31 | |
| BYOL-APre-training dataset=BYOL-A2022.06 | 62.6 | 62.3 | 78.8 | 89.6 | 92.4 | 13.7 | 39 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid BYOL-S/CvTalpha:beta ratio=1:42022.06 | 60.6 | 62.9 | 84.4 | 86.2 | 90.7 | 12 | 27.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-SEncoder=CLSTM2022.06 | 60.4 | 64 | 78.1 | 91.3 | 92.5 | 11.9 | 24.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-SEncoder=Resnetish-342022.06 | 60.3 | 63.5 | 77 | 83.2 | 88.9 | 13.3 | 35.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BYOL-S/CvT + OS2022.06 | 59.9 | 65.4 | 82.7 | 77.6 | 86.9 | 17.4 | 29.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| wav2vec2type=HEAR baseline2022.06 | 59.7 | 65.7 | 67.6 | 79.7 | 88.5 | 7.2 | 49.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| openSMILE (OS) only2022.06 | 52.4 | 59.4 | 66.7 | 70.1 | 80.2 | 13.1 | 25.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CREPEtype=HEAR baseline2022.06 | 23.8 | 36.2 | 49.9 | 16.8 | 19.6 | 5.1 | 15.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ATST-ClipClassifier=Shallow MLP, Embedding mode=Frozen, Toolkit=official hear-eval-kit2023.06 | — | 76 | 78.2 | 93.1 | 95.5 | 18.5 | — | 58.3 | 95.3 | 93.7 | 91.2 | 59.5 | 98.8 | 87.7 | 99.2 | 18.9 | 97.7 | 96.7 | 67.8 | 53.9 | — | — | — | — | — | — | |
| ATST-FrameClassifier=Shallow MLP, Embedding mode=Frozen, Toolkit=official hear-eval-kit, Frame-level handling=Concatenate outputs of all blocks2023.06 | — | 76.7 | 78.1 | 92.6 | 95.1 | 22.3 | — | 64.6 | 95.8 | 95.7 | 89 | 55.7 | 94.3 | 88.3 | 100 | 24.4 | 97.5 | 96.9 | 68.6 | 66.9 | — | — | — | — | — | — | |
| BestSource=HEAR leaderboard2023.06 | — | 75.2 | 78.5 | 97.6 | 97.8 | 21.5 | — | 87.8 | 97.5 | 92.5 | 96.1 | 64.1 | 96.7 | 90.8 | 99.2 | 46.9 | 97.5 | 94.1 | 87.8 | 72.2 | — | — | — | — | — | — | |
| CLAPEvaluation Protocol=MLP on frozen embeddings2023.04 | — | 64.36 | 77.83 | — | — | — | — | — | — | — | 96.7 | 58.59 | — | — | — | — | — | — | — | — | 96.83 | — | 100 | — | — | — | |
| CP-JKU PaSST 2lvl+melEvaluation Protocol=MLP on frozen embeddings2023.04 | — | 61.04 | 66.01 | — | — | 18.2 | — | — | — | — | 94.75 | 64.09 | — | — | — | — | — | — | — | — | 63.87 | 25.93 | 97.69 | — | — | — | |
| GURA Fuse HuBERTEvaluation Protocol=MLP on frozen embeddings2023.04 | — | 75.21 | 68.34 | — | — | 18.48 | — | — | — | — | 74.35 | 41.32 | — | — | — | — | — | — | — | — | 94.68 | 71.4 | 93.59 | — | — | — | |
| HEAR Leaderboard (SSL)Evaluation Protocol=MLP on frozen embeddings, Constraint=Self-supervised models only2023.04 | — | 75.21 | 78.53 | — | — | 18.48 | — | — | — | — | 80.5 | 50.88 | — | — | — | — | — | — | — | — | 96.87 | 71.4 | 96.86 | — | — | — | |
| HEAR Leaderboard BestEvaluation Protocol=MLP on frozen embeddings2023.04 | — | 75.21 | 78.53 | — | — | 22.69 | — | — | — | — | 96.65 | 65.48 | — | — | — | — | — | — | — | — | 97.79 | 72.02 | 99.23 | — | — | — | |
| LSSD (B.3)Representations=Audio-only, Evaluation Protocol=MLP on frozen embeddings2023.04 | — | 62.79 | 66.31 | — | — | 11.55 | — | — | — | — | 85.6 | 53.69 | — | — | — | — | — | — | — | — | 83.82 | 30.35 | 100 | — | — | — | |
| LSSD (B.4)Representations=Audio-only, Evaluation Protocol=MLP on frozen embeddings2023.04 | — | 63.18 | 68.88 | — | — | 10.82 | — | — | — | — | 83.75 | 51.61 | — | — | — | — | — | — | — | — | 77.12 | 28.19 | 100 | — | — | — | |
| LSSD (B.5)Representations=Audio-only, Evaluation Protocol=MLP on frozen embeddings2023.04 | — | 63.27 | 69.16 | — | — | 11.38 | — | — | — | — | 85.25 | 52.48 | — | — | — | — | — | — | — | — | 76.99 | 27.98 | 100 | — | — | — |