Universal Speech Representation Evaluation on SUPERB Benchmark
1,010.29Overall ScoreSpeech-FT
Evaluation results
| Method | Links | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Speech-FTFine-tuning Task=ER on IEMOCAP2025.02 | 1,010.29 | 0.8263 | — | — | — | — | — | — | — | — | — | 87.23 | 85.95 | — | — | — | — | — | — | 25.8 | 4.95 | |
| Speech-FTModel=WavLM Base+, Fine-tuning task=ASR on TED-LIUM2025.02 | 950.6 | 0.8812 | 4.34 | 3.84 | — | 5.18 | — | 97.44 | 0.109 | — | 99.1 | 90.73 | 69.02 | — | — | — | — | — | — | 21.1 | 3.42 | |
| Pre-trainedModel=WavLM Base+, Fine-tuning task=ASR on TED-LIUM2025.02 | 946.95 | 0.8943 | 4.12 | 3.5 | — | 5.57 | — | 97.36 | 0.0989 | — | 98.99 | 90.52 | 68.67 | — | — | — | — | — | — | 21.3 | 3.94 | |
| Speech-FTModel=WavLM Base+, Fine-tuning Task=ASR on TED-LIUM2025.02 | 945.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Pre-trainedModel=WavLM Base+2025.02 | 942.96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Stable-FTFine-tuning Task=ER on IEMOCAP2025.02 | 908.86 | 0.7307 | — | — | — | — | — | — | — | — | — | 81.4 | 90.07 | — | — | — | — | — | — | 36.62 | 16.11 | |
| Stable-FTModel=WavLM Base+, Fine-tuning Task=ASR on TED-LIUM2025.02 | 907.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Speech-FTFine-tuning Task=ASR on TED-LIUM2025.02 | 905.79 | 0.8411 | — | — | — | — | — | — | — | — | — | 88.84 | 67.78 | — | — | — | — | — | — | 23.46 | 3.94 | |
| Stable-FTModel=WavLM Base+, Fine-tuning task=ASR on TED-LIUM2025.02 | 905.09 | 0.832 | 5.04 | 5.19 | — | 6.29 | — | 96.72 | 0.1189 | — | 98.79 | 89.41 | 67.63 | — | — | — | — | — | — | 22.95 | 3.93 | |
| Speech-FTFine-tuning Task=PR on Librispeech2025.02 | 883.93 | 0.8195 | — | — | — | — | — | — | — | — | — | 88.6 | 65.79 | — | — | — | — | — | — | 24.52 | 2.99 | |
| Speech-FTFine-tuning Task=ASR on Librispeech2025.02 | 882.44 | 0.8279 | — | — | — | — | — | — | — | — | — | 87.97 | 66.38 | — | — | — | — | — | — | 24.93 | 3.93 | |
| Speech-FTFine-tuning Task=SID on Voxceleb12025.02 | 881.07 | 0.888 | — | — | — | — | — | — | — | — | — | 87.2 | 66.03 | — | — | — | — | — | — | 25.23 | 6.91 | |
| Speech-FTFine-tuning Task=SID on Librispeech2025.02 | 877.97 | 0.8324 | — | — | — | — | — | — | — | — | — | 87.98 | 65.48 | — | — | — | — | — | — | 24.9 | 5.67 | |
| Speech-FTFine-tuning Task=PC on TIMIT2025.02 | 877.66 | 0.8178 | — | — | — | — | — | — | — | — | — | 88.65 | 65.48 | — | — | — | — | — | — | 24.05 | 4.76 | |
| Speech-FTModel=HuBERT, Fine-tuning task=ASR on TED-LIUM2025.02 | 874.51 | 0.8411 | 5.27 | 5.99 | — | 5.75 | — | 96.69 | 0.1011 | — | 98.97 | 88.84 | 67.78 | — | — | — | — | — | — | 23.46 | 3.94 | |
| Speech-FTFine-tuning Task=ER on CREMA-D2025.02 | 871.93 | 0.8336 | — | — | — | — | — | — | — | — | — | 88.41 | 65.59 | — | — | — | — | — | — | 24.75 | 5.02 | |
| Stable-FTFine-tuning Task=ASR on TED-LIUM2025.02 | 870.71 | 0.7989 | — | — | — | — | — | — | — | — | — | 88.52 | 65.38 | — | — | — | — | — | — | 23.8 | 5.22 | |
| Pre-trainedFine-tuning Task=None2025.02 | 870.2 | 0.8186 | — | — | — | — | — | — | — | — | — | 88.54 | 64.99 | — | — | — | — | — | — | 24.7 | 5.17 | |
| Speech-FTModel=wav2vec 2.0, Fine-tuning Task=ASR on TED-LIUM2025.02 | 864.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Stable-FTFine-tuning Task=SID on Librispeech2025.02 | 847.33 | 0.7552 | — | — | — | — | — | — | — | — | — | 84.73 | 59.51 | — | — | — | — | — | — | 31.54 | 14.18 | |
| Stable-FTModel=HuBERT, Fine-tuning task=ASR on TED-LIUM2025.02 | 845 | 0.7989 | 5.34 | 6.02 | — | 7.63 | — | 96.92 | 0.097 | — | 98.71 | 88.52 | 65.38 | — | — | — | — | — | — | 23.8 | 5.22 | |
| Stable-FTFine-tuning Task=PR on Librispeech2025.02 | 842.03 | 0.756 | — | — | — | — | — | — | — | — | — | 88.09 | 63.14 | — | — | — | — | — | — | 26.24 | 2.21 | |
| Pre-trainedModel=wav2vec 2.02025.02 | 837.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Stable-FTModel=wav2vec 2.0, Fine-tuning Task=ASR on TED-LIUM2025.02 | 824.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mamba + MLPSize=Base, Parameters (M)=94.7, Setting=Causal2025.06 | 823.15 | 0.7348 | — | — | 11.72 | — | — | — | — | — | 89.62 | — | 61.72 | — | — | — | — | — | — | — | — | |
| MambaSize=Base, Parameters (M)=78.2, Setting=Causal2025.06 | 817.94 | 0.7307 | — | — | 11.68 | — | — | — | — | — | 87.45 | — | 61.98 | — | — | — | — | — | — | — | — | |
| Pre-trainedModel=HuBERT, Fine-tuning task=ASR on TED-LIUM2025.02 | 815.47 | 0.8186 | 5.56 | 6.32 | — | 6.38 | — | 96.59 | 0.0687 | — | 98.02 | 88.54 | 64.99 | — | — | — | — | — | — | 24.7 | 5.17 | |
| Causal TransformerSize=Base, Parameters (M)=94.7, Setting=Causal2025.06 | 805.44 | 0.6004 | — | — | 13.87 | — | — | — | — | — | 94.23 | — | 63.33 | — | — | — | — | — | — | — | — | |
| Stable-FTFine-tuning Task=ASR on Librispeech2025.02 | 805.33 | 0.7214 | — | — | — | — | — | — | — | — | — | 86.01 | 63.82 | — | — | — | — | — | — | 29.32 | 5.35 | |
| Speech-FTModel=DeCoAR 2.0, Fine-tuning Task=ASR on TED-LIUM2025.02 | 781.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mamba + MLPSize=Small, Parameters (M)=33.0, Setting=Causal2025.06 | 777.68 | 0.6842 | — | — | 15.28 | — | — | — | — | — | 82.39 | — | 61.63 | — | — | — | — | — | — | — | — | |
| Causal TransformerSize=Small, Parameters (M)=23.5, Setting=Causal2025.06 | 767.7 | 0.5827 | — | — | 15.91 | — | — | — | — | — | 87.92 | — | 62.07 | — | — | — | — | — | — | — | — | |
| MambaSize=Small, Parameters (M)=23.5, Setting=Causal2025.06 | 766.94 | 0.6606 | — | — | 17.34 | — | — | — | — | — | 85.13 | — | 60.82 | — | — | — | — | — | — | — | — | |
| Stable-FTFine-tuning Task=ER on CREMA-D2025.02 | 741.19 | 0.8148 | — | — | — | — | — | — | — | — | — | 87.28 | 64.69 | — | — | — | — | — | — | 27.03 | 6.62 | |
| Pre-trainedModel=DeCoAR 2.02025.02 | 736.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer + Causal MaskSize=Small, Parameters (M)=23.5, Setting=Causal2025.06 | 735.49 | 0.6661 | — | — | 24.05 | — | — | — | — | — | 81.86 | — | 60.45 | — | — | — | — | — | — | — | — | |
| Transformer + Causal MaskSize=Base, Parameters (M)=94.7, Setting=Causal2025.06 | 734.68 | 0.7457 | — | — | 25.57 | — | — | — | — | — | 80.07 | — | 58.77 | — | — | — | — | — | — | — | — | |
| Stable-FTFine-tuning Task=PC on TIMIT2025.02 | 726.64 | 0.6634 | — | — | — | — | — | — | — | — | — | 83.5 | 61.25 | — | — | — | — | — | — | 33.82 | 10.34 | |
| Stable-FTModel=DeCoAR 2.0, Fine-tuning Task=ASR on TED-LIUM2025.02 | 701.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Stable-FTFine-tuning Task=SID on Voxceleb12025.02 | 651.42 | 0.8775 | — | — | — | — | — | — | — | — | — | 77.64 | 60.29 | — | — | — | — | — | — | 43.27 | 29.08 | |
| WavLM Large#Params=316.62M, Corpus=Mix 94k hr, Weights=Frozen2021.10 | 74.6 | 95.49 | 3.77 | 3.24 | 3.06 | 3.44 | 32.27 | 97.86 | 0.0886 | 26.57 | 99.31 | 92.21 | 18.36 | 2.7 | 94.5 | 11.19 | 7.3 | 9 | 99 | — | — | |
| WavLM Base+#Params=94.70M, Corpus=Mix 94k hr, Weights=Frozen2021.10 | 73.4 | 89.42 | 4.07 | 3.5 | 3.92 | 5.59 | 38.32 | 97.37 | 0.0988 | 24.25 | 99 | 90.58 | 21.2 | 2.63 | 94.3 | 10.85 | 7.4 | 8.1 | 99 | — | — | |
| HuBERT Large#Params=316.61M, Corpus=LL 60k hr, Weights=Frozen2021.10 | 72.2 | 90.33 | 5.98 | 5.75 | 3.53 | 3.62 | 44.08 | 95.29 | 0.0353 | 20.01 | 98.76 | 89.81 | 21.76 | 2.64 | 94.2 | 10.45 | 7.22 | 9 | 99.25 | — | — | |
| WavLM Base#Params=94.70M, Corpus=LS 960 hr, Weights=Frozen2021.10 | 72 | 84.51 | 4.69 | 4.55 | 4.84 | 6.21 | 42.81 | 96.79 | 0.087 | 20.74 | 98.63 | 89.38 | 22.86 | 2.58 | 94 | 10.37 | 7.42 | 8 | 98 | — | — | |
| WavLM Base (w/o structure modification)#Params=94.68M, Corpus=LS 960 hr, Weights=Frozen2021.10 | 71.9 | 84.74 | 4.61 | 4.72 | 5.22 | 6.8 | 42.88 | 96.79 | 0.0956 | 20.03 | 98.31 | 88.56 | 24 | 2.58 | 94 | 10.29 | 7.45 | 8.4 | 99 | — | — | |
| WavLM Base (w/o denoising task)#Params=94.70M, Corpus=LS 960 hr, Weights=Frozen2021.10 | 71.7 | 84.39 | 4.91 | 6.03 | 4.85 | 6.08 | 43.61 | 96.79 | 0.0799 | 21.03 | 98.42 | 88.69 | 23.43 | 2.56 | 93.9 | 9.91 | 7.43 | 7.5 | 97.75 | — | — | |
| HuBERT Base#Params=94.68M, Corpus=LS 960 hr, Weights=Frozen2021.10 | 70.9 | 81.42 | 5.11 | 5.88 | 5.41 | 6.42 | 46.69 | 96.3 | 0.0736 | 15.53 | 98.34 | 88.53 | 25.2 | 2.58 | 93.9 | 9.36 | 7.47 | 8 | 98.5 | — | — | |
| wav2vec 2.0 Large#Params=317.38M, Corpus=LL 60k hr, Weights=Frozen2021.10 | 70.4 | 86.14 | 5.65 | 5.62 | 4.75 | 3.75 | 44.69 | 96.66 | 0.0489 | 12.48 | 95.28 | 87.11 | 27.31 | 2.52 | 94 | 10.02 | 7.63 | 15.8 | 97.25 | — | — | |
| wav2vec 2.0 Base#Params=95.04M, Corpus=LS 960 hr, Weights=Frozen2021.10 | 69.6 | 75.18 | 6.02 | 6.08 | 5.74 | 6.43 | 46.95 | 96.23 | 0.0233 | 14.81 | 92.35 | 88.3 | 24.77 | 2.55 | 93.9 | 9.77 | 7.5 | 10.5 | 98 | — | — | |
| DeCoAR 2.0#Params=89.84M, Corpus=LS 960 hr, Weights=Frozen2021.10 | 66.3 | 74.42 | 7.16 | 6.59 | 14.93 | 13.02 | 53.62 | 94.48 | 0.0406 | 9.94 | 90.8 | 83.28 | 34.73 | 2.47 | 93.2 | 8.54 | 7.83 | 17.1 | 90.75 | — | — | |
| wav2vec#Params=32.54M, Corpus=LS 960 hr, Weights=Frozen2021.10 | 63.5 | 56.56 | 7.99 | 9.9 | 31.58 | 15.86 | 55.86 | 95.59 | 0.0485 | 6.61 | 84.92 | 76.37 | 43.71 | 2.53 | 93.8 | 9.3 | 7.45 | 10.1 | 98.25 | — | — | |
| vq-wav2vec#Params=34.15M, Corpus=LS 960 hr, Weights=Frozen2021.10 | 61.8 | 38.8 | 10.38 | 9.93 | 33.48 | 17.71 | 60.66 | 93.38 | 0.041 | 5.66 | 85.68 | 77.68 | 41.54 | 2.48 | 93.6 | 8.16 | 7.08 | 13.4 | 100 | — | — | |
| VQ-APC#Params=4.63M, Corpus=LS 360 hr, Weights=Frozen2021.10 | 59.5 | 60.15 | 8.72 | 10.45 | 41.08 | 21.2 | 63.56 | 91.11 | 0.0251 | 4.23 | 74.48 | 68.53 | 52.91 | 2.56 | 93.4 | 8.44 | 7.84 | 22.4 | 94.25 | — | — | |
| APC#Params=4.11M, Corpus=LS 360 hr, Weights=Frozen2021.10 | 59.2 | 60.42 | 8.56 | 10.53 | 41.98 | 21.28 | 63.12 | 91.01 | 0.031 | 5.95 | 74.69 | 70.46 | 50.89 | 2.56 | 93.4 | 8.92 | 8.05 | 27.2 | 87.25 | — | — | |
| NPC#Params=19.38M, Corpus=LS 360 hr, Weights=Frozen2021.10 | 59 | 55.92 | 9.4 | 9.34 | 43.81 | 20.2 | 61.66 | 88.96 | 0.0246 | 4.32 | 69.44 | 72.79 | 48.44 | 2.52 | 93.1 | 8.04 | 7.86 | 30.4 | 94.75 | — | — | |
| TERA#Params=21.33M, Corpus=LS 960 hr, Weights=Frozen2021.10 | 57.2 | 57.57 | 15.89 | 9.96 | 49.17 | 18.17 | 58.49 | 89.48 | 0.0013 | 5.66 | 58.42 | 67.5 | 54.17 | 2.54 | 93.6 | 10.19 | 8.21 | 25.1 | 83.75 | — | — | |
| modified CPC#Params=1.84M, Corpus=LL 60k hr, Weights=Frozen2021.10 | 56.9 | 39.63 | 12.86 | 10.38 | 42.54 | 20.18 | 62.54 | 91.88 | 0.0326 | 4.82 | 64.09 | 71.19 | 49.91 | 2.57 | 93.7 | 10.4 | 8.41 | 26.2 | 71 | — | — | |
| PASE+#Params=7.83M, Corpus=LS 50 hr, Weights=Frozen2021.10 | 51.5 | 37.99 | 11.61 | 8.68 | 58.87 | 25.11 | 61.56 | 82.54 | 0.0072 | 3.16 | 29.82 | 62.14 | 60.17 | 2.56 | 93.9 | 9.87 | 8.66 | 30.6 | 63.2 | — | — | |
| Mockingjay#Params=85.12M, Corpus=LS 360 hr, Weights=Frozen2021.10 | 51 | 32.29 | 11.66 | 10.54 | 70.19 | 22.82 | 65.27 | 83.67 | 0.0007 | 4.45 | 34.33 | 61.59 | 58.89 | 2.53 | 93.4 | 9.29 | 8.29 | 35.1 | 79.75 | — | — | |
| FBANK#Params=0, Corpus=-, Weights=Frozen2021.10 | 43.2 | 0.0009 | 9.56 | 10.05 | 82.01 | 23.18 | 63.58 | 8.63 | 0.0058 | 2.32 | 9.1 | 69.64 | 52.94 | 2.55 | 93.6 | 9.23 | 8.47 | 38.3 | 77.25 | — | — | |
| HubertSize=Base2026.01 | — | 0.8142 | 5.11 | 5.88 | 5.41 | 6.42 | — | 96.3 | 0.0736 | — | 98.34 | 88.53 | 64.92 | — | — | — | — | — | — | 25.2 | — | |
| Mimi2026.01 | — | 0.3423 | 12.17 | 13.37 | 39.69 | 88.65 | — | 93.12 | 0.0108 | — | 90.61 | 6.72 | 56.05 | — | — | — | — | — | — | 98.86 | — | |
| Spectral (FBANK)2026.01 | — | 0 | 9.56 | 10.05 | 82.01 | 23.18 | — | 8.63 | 0.0058 | — | 9.1 | 69.64 | 35.39 | — | — | — | — | — | — | 52.94 | — | |
| SylberNotes=English-only original version2026.01 | — | 0.5124 | 8.75 | 15.55 | 88.79 | 8.88 | — | 97.11 | 0.0591 | — | 99.08 | 85.66 | 65.25 | — | — | — | — | — | — | 29.49 | — | |
| Sylber 2.0Notes=Universal version2026.01 | — | 0.7203 | 5.69 | 6.1 | 14.74 | 10.04 | — | 96.85 | 0.0471 | — | 98.05 | 85.94 | 62.37 | — | — | — | — | — | — | 30.09 | — | |
| Wav2Vec 2.0Size=Base2026.01 | — | 0.7518 | 6.02 | 6.08 | 5.74 | 6.43 | — | 96.23 | 0.0233 | — | 92.35 | 88.3 | 63.43 | — | — | — | — | — | — | 24.77 | — | |
| WavLMSize=Large2026.01 | — | 0.9549 | 3.77 | 3.24 | 3.06 | 3.44 | — | 97.86 | 0.0886 | — | 99.31 | 92.21 | 70.62 | — | — | — | — | — | — | 18.36 | — |